
databricks
Databricks skills for the CLI, Apps, Lakebase, Model Serving, Lakeflow Jobs, Spark Declarative Pipelines, Declarative Automation Bundles (DABs), and classic-to-serverless migration.
- GitHub stars
- 404
Install
/plugin install databricks@databricks-agent-skillsRun it inside Claude Code after adding the marketplace with /plugin marketplace add databricks/databricks-agent-skills.
Contents
Discovered from sourceSkills 38
Databricks AI Functions
Official Docs: https://docs.databricks.com/large-language-models/ai-functions Individual function reference: https://docs.databricks.com/sql/language-manual/sql-ref-functions-builtin
Overview
Databricks AI Functions are built-in SQL and PySpark functions that call Foundation Model APIs directly from your data pipelines — no model endpoint setup, no API keys, no boilerplate. They operate on table columns as naturally as UPPER() or LENGTH(), and are optimized for batch inference at scale.
Always prefer a task-specific function over ai_query. Reach for ai_query only when no task function fits (custom/external endpoints, multimodal, or JSON beyond ai_extract's limits). Every function below shares a baseline: DBR 15.1+ (notebooks) / 15.4 ML LTS (batch), not on SQL Warehouse Classic, and region must support AI Functions — the Prereqs column lists only what's additional.
Cost & speed — each call is an LLM inference (slow and billed per token). Run a function once per row and persist the result to a Delta table; never re-invoke it on every downstream query. In demos, avoid generating tables with millions of rows — sample the input when needed so the demo runs quickly. Materialize once, then query the cheap Delta output.
The Function column links to the in-repo deep reference (full options, schemas, examples); Docs links to the official page.
| Function | Task | Input | Output | Extra prereqs | Docs |
|---|---|---|---|---|---|
ai_analyze_sentiment | Sentiment scoring | content STRING | STRING — positive/negative/neutral/mixed, or NULL | — | ↗ |
ai_classify | Fixed-label routing | content STRING|VARIANT, labels (2–500), [options MAP] | v2.1 VARIANT — {response:[{value:label}], error_message, metadata} | — | ↗ |
ai_decide | Evaluate questions (boolean / choice / score) over text or structured data | state STRING|VARIANT, questions STRING (JSON), [options MAP] | VARIANT — {response:{answers}, metadata, error_message} | Beta; admin-gated via Previews | ↗ |
ai_extract | Entity / field extraction | content STRING|VARIANT, schema STRING (JSON), [options MAP] | v2.1 VARIANT — {response:{field:{value:…}}, error_message, metadata} | ≤256 fields, ≤12 nesting levels | ↗ |
ai_fix_grammar | Grammar correction | content STRING | STRING (corrected) | — | ↗ |
ai_gen | Free-form generation | prompt STRING | STRING | — | ↗ |
ai_mask | PII redaction | content STRING, labels ARRAY<STRING> | STRING (entities → [MASKED]) | — | ↗ |
ai_similarity | Semantic similarity | expr1 STRING, expr2 STRING | FLOAT (0.0–1.0) | — | ↗ |
ai_summarize | Summarization | content STRING, [max_words INT] (0 = uncapped) | STRING | Public Preview; English-tuned | ↗ |
ai_translate | Translation | content STRING, to_lang STRING | STRING | Langs: en, fr, de, hi, it, pt, es, th | ↗ |
ai_parse_document | Parse PDF / Office / images | content BINARY, [Map('version','2.0', …)] | VARIANT — pages, elements, error_status | DBR 17.3+; ≤500 pages / 100 MB | ↗ |
ai_prep_search | RAG chunking from parsed docs or text | parsed VARIANT|STRING, [options MAP] | VARIANT — {document:{contents, pages, source_uri}, error_status} | DBR 18.2+ (serverless env v3+) | ↗ |
ai_query | Hosted model service or custom serving endpoint, multimodal, complex JSON (last resort) | endpoint STRING, request STRING|STRUCT, [returnType], [failOnError BOOL], [modelParameters STRUCT], [responseFormat STRING], [files] | Parsed response; with failOnError => false a STRUCT{response, errorMessage} | Pro/Serverless warehouse; EXECUTE on model service or CAN QUERY on serving endpoint | ↗ |
ai_forecast | Time series forecasting (table-valued) | observed TABLE, horizon, time_col, value_col, [group_col], [prediction_interval_width], [frequency], [seed], [parameters] | Rows: time/group cols + per value {v}_forecast, {v}_upper, {v}_lower (DOUBLE) | Pro/Serverless warehouse; Public Preview | ↗ |
Models run under Apache 2.0 or LLAMA 3.3 Community License — you are responsible for compliance.
Patterns
Pin extraction and classification to v2.1 in new code. Every extracted scalar and returned label has a value field, even when citations, confidence scores, and rationales are disabled. Existing v2.0 queries remain supported; update downstream projections to read value when upgrading. See the task-function reference for the version-specific contracts.
Chain task functions to enrich a column in one pass. ai_classify/ai_extract return a VARIANT — read it with the colon operator (:response):
SELECT id,
ai_analyze_sentiment(content) AS sentiment,
ai_summarize(content, 30) AS summary,
ai_classify(content, '["technical","billing","other"]', map('version','2.1')):response[0].value::STRING AS category,
ai_extract(content, '["product","error_code","date"]', map('version','2.1')):response:product:value::STRING AS product,
ai_fix_grammar(content) AS content_clean
FROM raw_feedback;
In PySpark, call any of these inside expr(...): df.withColumn("category", expr("ai_classify(content, '[\"a\",\"b\"]', map('version','2.1')):response[0].value::STRING")) — and read extracted scalar fields via selectExpr("col:response:field:value::STRING AS field").
PII redaction before storage — ai_mask(content, ARRAY(entity_types)) returns text with entities → [MASKED].
SELECT ai_mask(message, array('person','email','phone','address')) AS message_safe FROM raw_messages;
Semantic matching / dedup — ai_similarity returns 0–1; self-join and threshold:
SELECT a.id, b.id, ai_similarity(a.name, b.name) AS score
FROM companies a JOIN companies b ON a.id < b.id
WHERE ai_similarity(a.name, b.name) > 0.85;
Forecasting — table-valued; one row per future period (+ per group). Full param/group/interval forms → 3-ai-forecast.md:
SELECT * FROM ai_forecast(
observed => TABLE(SELECT date, sales FROM daily_sales),
horizon => '2026-12-31', time_col => 'date', value_col => 'sales');
-- Returns: date, sales_forecast, sales_upper, sales_lower
Nested field extraction — ai_extract supports arrays of objects, including invoice line items. See extraction schemas and versioned outputs.
Document parsing (ai_parse_document) and RAG chunking (ai_prep_search) get their own staged pipeline below.
Document Processing Pipeline
Chain AI Functions stage-by-stage into Delta tables for batch document processing. The example is written as a Spark Declarative Pipeline (SDP / Lakeflow / DLT) — CREATE OR REFRESH STREAMING TABLE with STREAM(...) sources. To run the same logic standalone in a notebook / SQL warehouse, swap each CREATE OR REFRESH STREAMING TABLE x AS for CREATE OR REPLACE TABLE x AS and drop the STREAM(...) wrappers. In SDP Python it's @dp.table with from pyspark import pipelines as dp.
Pass the parsed VARIANT directly into extraction and classification to preserve layout and source context. For plain-text or Markdown files, skip parsing: use read_files(..., format => 'text', wholetext => true) and pass its value column. Without wholetext => true, text input is read one line per row.
-- Stage 1 — parse supported PDF, image, or Office documents, filter parse errors
CREATE OR REFRESH STREAMING TABLE raw_parsed AS
SELECT path,
parsed AS parsed_content,
parsed:error_status AS parse_error
FROM (
SELECT path, ai_parse_document(content, map('version','2.0')) AS parsed
FROM STREAM read_files('/Volumes/my_catalog/doc_processing/landing/', format => 'binaryFile')
)
WHERE is_variant_null(parsed:error_status);
-- Stage 2 — classify document type (cheap, no endpoint selection)
CREATE OR REFRESH STREAMING TABLE classified_docs AS
SELECT *,
ai_classify(parsed_content, '["invoice","purchase_order","receipt","contract","other"]', map('version','2.1')):response[0].value::STRING AS doc_type
FROM STREAM raw_parsed;
-- Stage 3 — extract fields; ai_extract returns a VARIANT, read fields with `:`
CREATE OR REFRESH STREAMING TABLE extracted AS
SELECT path, doc_type,
result:response:invoice_number:value::STRING AS invoice_number,
result:response:vendor_name:value::STRING AS vendor_name,
result:response:total_amount:value::DOUBLE AS total_amount,
result:error_message::STRING AS extract_error
FROM (
SELECT *, ai_extract(parsed_content,
'{"invoice_number":{"type":"string"},"vendor_name":{"type":"string"},"total_amount":{"type":"number"}}',
map('version','2.1')) AS result
FROM STREAM classified_docs WHERE doc_type = 'invoice' AND parsed_content IS NOT NULL
);
In a batch job, retain per-row errors for inspection or route them to a sidecar table: check ai_extract's result:error_message with is_variant_null for JSON null on success, and for ai_query pass failOnError => false and check ai_response.errorMessage (a STRUCT field, dot-accessed). SQL IS NULL does not recognize a JSON null inside VARIANT. See 2-ai-query.md.
Custom RAG Pipeline — Parse → Prep → Index
For retrieval rather than field extraction: ai_parse_document → ai_prep_search (semantic chunking + context enrichment, DBR 18.2+) → Vector Search Delta Sync index. ai_prep_search returns chunk_id, chunk_to_retrieve, and chunk_to_embed (enriched with title/headers/page) — embed chunk_to_embed, return chunk_to_retrieve to the LLM. Shown standalone; in an SDP swap CREATE OR REPLACE TABLE for CREATE OR REFRESH STREAMING TABLE + STREAM read_files(...).
CREATE OR REPLACE TABLE parsed_chunks AS
WITH prepped AS (
SELECT path AS source_path, ai_prep_search(ai_parse_document(content, map('version','2.0'))) AS prep
FROM read_files('/Volumes/my_catalog/doc_processing/docs/', format => 'binaryFile')
)
SELECT
variant_get(chunk, '$.chunk_id', 'STRING') AS chunk_id,
variant_get(chunk, '$.chunk_to_retrieve', 'STRING') AS chunk_to_retrieve,
variant_get(chunk, '$.chunk_to_embed', 'STRING') AS chunk_to_embed,
source_path
FROM prepped LATERAL VIEW explode(variant_get(prep, '$.document.contents', 'ARRAY<VARIANT>')) c AS chunk;
Then enable CDF (ALTER TABLE parsed_chunks SET TBLPROPERTIES (delta.enableChangeDataFeed = true)) and use the databricks-vector-search skill to build a Delta Sync index: PK chunk_id, embedding source chunk_to_embed, return chunk_to_retrieve.
Beyond batch:
- Ask questions over the output — point a Genie Agent at the resulting Delta table for natural-language querying instead of hand-written SQL; see the databricks-genie-agents skill.
- Low-latency / serving — to expose this as a real-time, governed endpoint (e.g. register a model to Unity Catalog and serve it), use the databricks-model-serving skill.
- Production incremental ingestion — for a runnable end-to-end streaming
ai_parse_documentjob (checkpoints,trigger(availableNow=True)), see databricks/bundle-examples · job_with_ai_parse_document.
Reference Files
- 1-task-functions.md — Deep reference for every task-specific function: full options/schemas (e.g.
ai_extractv2.1 citations + confidence scores,ai_classifymultilabel,ai_parse_documentoptions + output schema,ai_prep_searchchunk schema) and non-trivial examples. The Overview table above links to each function's section directly. - 2-ai-query.md —
ai_querycomplete reference: all parameters, structured output withresponseFormat, multimodalfiles =>, UDF patterns, and error handling - 3-ai-forecast.md —
ai_forecastparameters, single-metric, multi-group, multi-metric, and confidence interval patterns
Common Issues
| Issue | Solution |
|---|---|
ai_parse_document not found | Requires DBR 17.3+. Check cluster runtime. |
ai_prep_search not found | Requires DBR 18.2+ (serverless env v3+). |
ai_decide not found / unavailable | Beta — a workspace admin must enable it on the Previews page. Requires Databricks SQL (not SQL Classic) or DBR 15.4 LTS+. |
explode() fails on a VARIANT | explode needs ARRAY — cast first: explode(variant_get(prep, '$.document.contents', 'ARRAY<VARIANT>')). |
| Embedding the wrong RAG column | Embed chunk_to_embed (context-enriched); return chunk_to_retrieve to the LLM. |
ai_forecast fails | Requires Pro or Serverless SQL warehouse — not available on Classic or Starter. |
| All functions return NULL | Input column is NULL. Filter with WHERE col IS NOT NULL before calling. |
ai_translate fails for a language | Supported (8): English (en), French (fr), German (de), Hindi (hi), Italian (it), Portuguese (pt), Spanish (es), Thai (th) — to_lang takes the code or the full name (see references/1-task-functions.md). Use ai_query with a multilingual model for others. |
ai_classify returns unexpected labels | Use clear, mutually exclusive label names. Fewer labels (2–5) produces more reliable results. |
ai_query raises on some rows in a batch job | Add failOnError => false — returns a STRUCT with .response and .errorMessage (dot-accessed) instead of raising. |
ai_query fails with 403 "no longer available" | Discover the exact replacement system.ai model service using the AI Gateway listing; legacy names do not always have a matching model service. Custom serving endpoints keep their existing names. See 2-ai-query.md. |
| Batch job runs slowly | Use DBR 15.4 ML LTS cluster (not serverless or interactive) for optimized batch inference throughput. |
Databricks AI Runtime (databricks air)
Databricks AI Runtime is a set of databricks air CLI commands for submitting GPU training workloads to Databricks serverless compute. It manages environment setup, distributed training configuration, and workload lifecycle, without requiring you to manage clusters or infrastructure.
This skill covers the three everyday flows: starting a run, checking on a run, and monitoring a run as it trains.
Reference discipline
The CLI's built-in help is the source of truth for field names, accelerator
types, defaults, and constraints. Prefer it at runtime over anything memorized.
Look up a config field by passing its path to -h on the run command (the
path is a separate argument):
databricks air run -h config # list the YAML fields
databricks air run -h config.compute # one field
databricks air run -h config.compute.accelerator_type # a nested field
databricks air <subcommand> --help documents each command's flags. This skill
covers workflow and the few things the help does not spell out.
Session setup
Check the CLI version, confirm the command is available, and list your Databricks config profiles:
databricks version
databricks air --help
databricks auth profiles --skip-validate # available Databricks config profiles
The examples use flattened code_source fields, available in
Databricks CLI 1.20.0 and newer.
On an older CLI, use the equivalent nested snapshot format shown under
Workload YAML. CLI 1.20.0 and newer accept both formats.
Do not mix nested and flattened fields in the same configuration.
Pass -p <profile> on every command. Add -o json for scripted/programmatic
calls so you get a structured envelope instead of human text:
- Success:
{"v": 1, "ts": "...", "data": {...}} - Error:
{"v": 1, "ts": "...", "error": {"code": "...", "kind": "...", "message": "...", "retryable": bool}}
Config validation (input) errors are the exception: they print as plain text to
stderr with a non-zero exit even under -o json, so check the exit code and
stderr too, not just the parsed JSON. When you do get a structured error, its
kind tells you what to do:
| kind | Meaning | Action |
|---|---|---|
PERMANENT | Not authenticated / permission denied / feature disabled / non-retryable | If auth: databricks auth login --host <workspace-url> |
NOT_FOUND | Bad run ID / resource | Check the input |
CONFLICT | Idempotency key collision | Use a new --idempotency-key |
TRANSIENT | Retryable | Retry |
Subcommands
| Command | What it does |
|---|---|
databricks air run | Submit a workload from a YAML file. Add --watch to stream logs until it finishes, --dry-run to validate without submitting, and --override <field>=<value> (repeatable) to change a field without editing the YAML. |
databricks air get <JOB_RUN_ID> | Show status, config, timing, and the failure reason for one run. |
databricks air list | List your active runs. A run that just finished drops out of the default view; add --all-status to include finished runs. Also --all-users, --filter KEY=VALUE, --limit. |
databricks air cancel <JOB_RUN_ID> [<JOB_RUN_ID>...] / ... --all | Cancel one or more runs, or all of your active runs (--all prompts; add --yes in scripts). |
databricks air logs <JOB_RUN_ID> | Print a run's logs. Newer CLIs print the logs available now and exit, and stream with --follow; older CLIs (no --follow in --help) stream an active run by default. |
databricks air pools list / pools get <POOL_ID> | View the provisioned GPU pools available to the workspace. |
databricks air images push | Push a local or remote container image to Databricks Artifact Registry. See docker-images.md. |
databricks air convert-to-dabs <yaml_path> | Convert a workload YAML into a Databricks Asset Bundle (local only; deploy with databricks bundle deploy). |
For flags and defaults, run databricks air <subcommand> --help.
Workload YAML
experiment_name: my-training-job # alphanumeric, - and _ only
compute:
num_accelerators: 1
accelerator_type: GPU_1xA10 # run `databricks air run -h config.compute` for valid values
environment:
version: "6" # Standard environment; use databricks_ai_v6 for PyTorch preinstalled
dependencies:
- torch
- mlflow
code_source:
root_path: "." # relative to workload.yaml; extracted to $CODE_SOURCE_PATH on each node
max_retries: 0 # while iterating; see Retries below
command: |-
cd "$CODE_SOURCE_PATH"
python train.py
Submit with databricks air run --file workload.yaml -p <profile>. Field
details live in databricks air run -h config.<field> (the authoritative
source).
For an older CLI, replace only the code_source block above with:
code_source:
type: snapshot
snapshot:
root_path: "."
The values and behavior are the same: add type: snapshot under code_source
and put all snapshot fields (root_path, and any remote_volume, git, or
include_paths) inside code_source.snapshot. Keep the rest of the workload
unchanged. Check databricks air run -h config.code_source for the fields
supported by the installed CLI.
Accelerator types
Run databricks air run -h config.compute to see the accelerator types. On CLI
1.23.0 and newer it lists the types available in the current workspace and
how many GPUs each type has per node. If it cannot fetch them, it prints a
warning. Some older CLI versions also show a built-in fallback that may include
types the workspace can't use. Do not guess a type from memory: availability
depends on the workspace, cloud, and region, and matching is case-sensitive.
Type names follow GPU_<GPUs per node>x<GPU>, for example GPU_1xA10,
GPU_1xH100, GPU_8xH100, and GPU_8xB300. num_accelerators is the total
GPU count across all nodes, so it must be a multiple of the per-node count: 8
on an 8x type is one node, 16 is two nodes. GPU_1xH100 does not support
multi-node configurations, so set num_accelerators: 1.
Python environment
Pick the environment from what the training code imports, before you submit, so a missing package does not surface only after a GPU has been provisioned:
environment | What it provides | Use when |
|---|---|---|
version: "6" (the default) | Standard environment: Python and CUDA only. Since version 5, it does not include torch or torchvision. | You control the dependency stack. List every import, including torch, under dependencies. |
version: databricks_ai_v6 | Databricks AI environment: PyTorch, Transformers, and other ML frameworks preinstalled. | Typical PyTorch or Hugging Face training. Add extras under dependencies. |
unity_catalog_image: <catalog>.<schema>.<image>:<tag> | Your own Docker image. | System libraries or CUDA extensions; see Custom Docker images. |
Before submitting, check the script's imports against the chosen environment.
For the exact packages in each environment version, see the
environment version release notes.
Do not combine unity_catalog_image with version or dependencies.
Configuring the workload
These optional top-level fields cover the common needs. Run
databricks air run -h config.<field> for the full reference on any of them.
Hugging Face gated models
When a model needs a Hugging Face (HF) token, lead with this checklist before asking for other training details or running secrets commands:
- Accept the model license on HF and, if required, wait for access approval. Create a fine-grained read-only token with Read access to contents of all public gated repos you can access. For a private repo, also grant read access to that specific repo.
- Choose an existing scope the user owns and can write. If they need to find
one, have them filter
list-scopesto their own username or scopes they created, then confirm ownership and access; a name match alone is not proof. For example, the user can run the following with their own username:
Never browse or inspect other users' scopes or keys. Create a scope only if needed; ifdatabricks secrets list-scopes -p <profile> -o json | jq -r --arg user "<your-username>" '.[] | select(.name | contains($user)) | .name'create-scopereportsMax limit of 2000 scopes has been reached, reuse an existing user-owned scope instead. The user may need to runlist-aclson their own scope to check access. Before using<scope>in anydatabricks secrets ...command, validate it: it must be an existing scope the user owns, from their own scopes, and match the scope-name charset^[A-Za-z0-9_.@-]+$. If it doesn't, have the user run the commands themselves. Always double-quote it. If the scope is Azure Key Vault-backed (backend_typeisAZURE_KEYVAULT),put-secretcan't write to it: have the user add the token to that vault with their own tooling and use that secret's name as the key below. - First check whether
hf-tokenalready exists in the scope, using the key-name check in step 4. If it does, reuse it only if it is the right token; otherwise pick a new key such ashf-token-gemma, or get the user's explicit OK before overwriting it, since other workloads may read it. Use the chosen key in steps 4 and 5. Have the user run this in their own terminal and enter the token at the hidden prompt. Never paste the token into chat or a command line.databricks secrets put-secret "<scope>" hf-token -p <profile> - Once they have stored it and named their scope, verify only the key name in
that one scope, filtering the output to
hf-token:databricks secrets list-secrets "<scope>" -p <profile> -o json | jq -r '.[] | select(.key == "hf-token") | .key' - Reference
secrets: {HF_TOKEN: "<scope>/hf-token"}in the workload. Quote the reference, particularly when the scope contains@or.. Transformers readsHF_TOKENautomatically.
Gemma 3 is gated; Mistral Small 3.2, gpt-oss-20b, and Qwen3 are ungated and do not need a token just to download their public weights. Check the specific model's HF page for current access requirements.
Training data in Unity Catalog
When the training script reads rows from a Unity Catalog (UC) table, use
UCDataset (Beta), a PyTorch dataset that reads the table's files directly.
An AIR workload has no spark global, so code that calls spark.table() fails.
Lead with this checklist before writing the script:
- Use
environment.version: databricks_ai_v6and adddatabricks-uc-dataset==0.0.1b1toenvironment.dependencies. If the script also importsdatabricks.air.data, for example itsDataLoader, add"databricks-sdk-air[data]"as well, as the AIR docs ask of CLI workloads; somedatabricks_ai_v6builds don't include it. Withunity_catalog_image, which can't be combined withversionordependencies, install these packages in the image instead. - Launch with
torchrun, even on one GPU, for exampletorchrun --nproc_per_node=$LOCAL_WORLD_SIZE train.py. AIR setsWORLD_SIZEbut notRANK, andUCDatasetraises when only one of them is set. With DDP, calltorch.distributed.init_process_group()before you create the dataset, and create it on every rank. - The workload reads the table as the identity that submits it, which needs
USE CATALOG,USE SCHEMA, andSELECT.UCDatasetreads only Delta tables in UC. It can't read views, materialized views, streaming tables, shallow clones, tables with aVARIANTcolumn, or tables with row filters or column masks; see the limitations. - Before submitting, check each table, so a wrong name, a missing privilege, or
an unsupported table fails in seconds instead of after the GPUs start. Keep
the arguments in single quotes so the shell doesn't expand them:
Continue only ifdatabricks tables get '<catalog>.<schema>.<table>' -o json -p <profile> databricks experimental aitools tools query 'SELECT COUNT(*) FROM IDENTIFIER(:t)' --param 't=<catalog>.<schema>.<table>' -p <profile>table_typeisMANAGEDorEXTERNAL,data_source_formatisDELTA, there's norow_filter, no column has amaskor aVARIANTtype, and the count returns. Backtick-quote name parts that contain characters other than letters, digits, and underscores; if a name contains', have the user run the checks. The query needs a SQL warehouse (add--warehouse <id>to choose one), andexperimentalcommands can change between CLI versions. If the query can't run for a reason unrelated to the table, report the check as inconclusive, not as a missing privilege.
If creating the dataset fails with PERMISSION_DENIED: ... prohibited from retrieving temporary storage credentials from Unity Catalog in network restricted workspaces, the workspace's serverless network policy blocks
UCDataset. The checks in step 4 still pass in that case, and tables in the
samples catalog may still load. Retrying, or changing how the script calls
UCDataset, won't help: tell the user, since an account admin controls that
policy. If the user says the policy changed and the same error comes back,
resubmit at most once with max_retries: 0, then stop.
For files in a UC volume, such as images or audio, use
databricks.air.data.UCVolumeDataset instead. For a complete workload, see
Load UC tables into PyTorch with UCDataset
and the cookbook's
click-through-rate recipe.
Environment variables and secrets
Set plain values under env_variables; pull sensitive values from Databricks
Secrets under secrets (each entry maps an env var name to a scope/key, and
the value is materialized on the worker, never logged):
env_variables:
BATCH_SIZE: "32"
secrets:
HF_TOKEN: "<scope>/hf-token" # -> $HF_TOKEN on every node
WANDB_API_KEY: "<scope>/wandb_key"
The scope and keys must already exist. Prefer a user-owned scope; create one
only if needed, and have the user store secret values in their own terminal.
Secret env var names must not collide with env_variables names.
Parameters (hyperparameters)
parameters is the clean way to pass hyperparameters into your script and
change them between runs without editing training code. The CLI writes them to
a YAML file on the cluster and exposes its path as $HYPERPARAMETERS_PATH:
parameters:
learning_rate: 2e-5
batch_size: 32
warmup_steps: 1000
use_amp: true
Read them in the script, converting every number to the type your code expects:
import os, yaml
with open(os.environ["HYPERPARAMETERS_PATH"]) as f:
hp = yaml.safe_load(f)
lr = float(hp["learning_rate"])
batch_size = int(hp["batch_size"])
warmup_steps = int(hp["warmup_steps"])
use_amp = hp["use_amp"]
Always convert numeric values with float() or int(), including values in
nested dicts and lists: yaml.safe_load can return floats such as 2e-5,
0.00002, or 1e6 as strings. Write integers as plain digits (1000000,
not 1e6), because int("1e+06") fails.
Strings, booleans, lists, and nested dicts arrive with their types. Quote a
value that must stay a string but looks like a number (run_tag: "1e5",
version: "1.10"); unquoted, it is read as a number. To sweep a
hyperparameter without editing the YAML, resubmit with
--override parameters.learning_rate=0.0005 (and a fresh idempotency key).
Retries
Failed attempts are retried automatically: max_retries defaults to 3 when
unset (confirm with databricks air run -h config.max_retries). Each retry
provisions GPUs and reruns the whole command, so a deterministic failure (an
import error, a typo, a bad path) fails 4 times before the run gives up.
Set max_retries: 0 while writing, debugging, or iterating on a workload,
either in the YAML or with --override max_retries=0. Raise it only for a
long, stable training run where retrying can recover from transient
infrastructure failures, and make the training code resume from a checkpoint
so a retry does not start over. When timeout_minutes is set, it applies
per attempt.
Permissions
By default only you can see the run. Grant others access to the submitted job
with permissions (DABs-compatible format); levels are CAN_VIEW,
CAN_MANAGE_RUN, CAN_MANAGE:
permissions:
- group_name: data-engineering
level: CAN_VIEW
- user_name: alice@example.com
level: CAN_MANAGE
Other fields
| Field | Use |
|---|---|
timeout_minutes | Wall-clock limit per attempt, in minutes. |
idempotency_token | YAML equivalent of --idempotency-key: resubmitting with the same token returns the existing run. |
mlflow_run_name, mlflow_experiment_directory | Name the MLflow run, or choose the workspace directory that holds the experiment. |
compute.pool_id, compute.priority_class | Run on a pre-provisioned GPU pool (find IDs with databricks air pools list). priority_class is NORMAL or BEST_EFFORT (preemptable) and requires pool_id. |
containers | Run different images and commands on different node ranks, for example a trainer and an inference server. Replaces command; each entry needs name, command, ranks, and unity_catalog_image. See databricks air run -h config.containers. |
Environment variables provided on each node
databricks air sets these automatically on every node; use them in command:
and in your training code.
| Variable | Meaning |
|---|---|
CODE_SOURCE_PATH | Absolute path to your extracted code_source on the node (cd $CODE_SOURCE_PATH). |
MASTER_ADDR / MASTER_PORT | Address/port of the rank-0 node for distributed coordination. |
NUM_NODES | Total number of nodes. |
NODE_RANK | Rank of the current node (0 to NUM_NODES-1). Also set as POD_RANK. |
WORLD_SIZE | Total number of GPUs across all nodes. |
LOCAL_WORLD_SIZE | Number of GPUs on this node. |
MLFLOW_RUN_ID | MLflow run created for this attempt (log metrics/artifacts to it). mlflow.start_run() attaches to it. The Hugging Face Trainer's MLflow integration starts it automatically, so don't also call mlflow.start_run() around trainer.train(): it fails with "Run ... is already active". |
HYPERPARAMETERS_PATH | Path to the parameters YAML file (only set when parameters: is defined). |
Your command: runs once per node, so per-process ranks (RANK,
LOCAL_RANK) are not set by the platform. Your distributed launcher
(torchrun, accelerate) assigns them when it spawns one process per GPU; pass it
$NODE_RANK / $NUM_NODES / $MASTER_ADDR / $MASTER_PORT.
Starting a run
Start from a cookbook recipe when one fits. The
AI Runtime Cookbook
has runnable recipes, each a workload.yaml plus source code, grouped by domain
(language-models/, computer-vision/, recommendation-and-ranking/,
time-series/, multimodal/, serving/, and more; Ray recipes live in these
domain folders with a ray- prefix). getting-started/air-cli/ is the
minimal first run. Adapting the closest recipe is usually faster and less
error-prone than writing a workload from scratch:
git clone https://github.com/databricks-solutions/ai-runtime-cookbook.git
cd ai-runtime-cookbook/<domain>/<recipe> # e.g. getting-started/air-cli
# Review workload.yaml (and the recipe README) for values to configure, then:
databricks air run --file workload.yaml --dry-run -o json -p <profile>
databricks air run --file workload.yaml --watch \
--idempotency-key <key> -o json -p <profile> # key as in step 5
Apply the steps below to the recipe's YAML the same way as to a new one,
including the stable --idempotency-key from step 5.
-
Turn the user's intent into the YAML. Extract model, dataset path, GPU count and type, and the training command from the request. For gated models, lead with the token checklist first. For data in a UC table, follow the UC data checklist. Ask before synthesizing a config when a remaining critical detail is ambiguous (data path, checkpoint dir, custom image).
-
Write the
command:using the env vars above. Common launch shapes:Framework command:shapeplain torch, single GPU python train.py(withUCDataset, usetorchrun; see the UC data checklist)torch, multi-GPU (1 node) torchrun --nproc_per_node=$LOCAL_WORLD_SIZE train.pytorch, multi-node torchrun --nnodes=$NUM_NODES --nproc_per_node=$LOCAL_WORLD_SIZE --node_rank=$NODE_RANK --master_addr=$MASTER_ADDR --master_port=$MASTER_PORT train.pyHuggingFace Accelerate accelerate launch --config_file accelerate.yaml train.pyDeepSpeed deepspeed --num_gpus=$LOCAL_WORLD_SIZE train.pyComposer / MosaicML composer train.pyMulti-line commands go under
command: |-. Access uploaded code via$CODE_SOURCE_PATH. Total GPUs = nodes x accelerators-per-node; see Accelerator types. -
Pre-flight the referenced resources so the job does not die on startup:
- Every package the script imports is in the chosen
environment or its
dependencies(the Standard environment has notorch). max_retries: 0is set while the workload is new or being debugged.- For each
secretsreference, verify only its key name in the user's chosen scope, filteringlist-secretsas in the token checklist. Never browse other users' scopes or list their keys. - Every UC table the script reads passes both checks in the UC data checklist. A dry run doesn't check UC tables, however the script gets their names.
- If
environment.unity_catalog_imageis set, the image has been pushed to Databricks Artifact Registry in the same workspace and the submitting identity can read it (see docker-images.md).
- Every package the script imports is in the chosen
environment or its
-
Dry-run before submitting. This validates the config without launching:
databricks air run --file workload.yaml --dry-run -o json -p <profile>On CLI 1.20.0 and newer, the dry run also checks the config against the workspace, so it needs a working profile. A config-validation error is a hard stop: it prints as plain text on stderr with a non-zero exit, so show that message to the user and fix the YAML. Other failures are infra/auth and should bubble up.
-
Submit. Use a stable
--idempotency-keyso an accidental re-run does not double-submit (compose it from<user>-<experiment>-<YYYYMMDD>-v<N>and bumpvNon each resubmit; avoid dynamic subexpressions like$(uuidgen)so the command stays auto-approvable). If you want to wait for the run to finish or stream live progress, add--watch:databricks air run --file workload.yaml --watch \ --idempotency-key <key> -o json -p <profile>--watchstreams logs and status updates until the run finishes and exits 0 only if the run succeeds. With-o json, it emits JSONL and includes the run ID. For long runs, use the agent's background-execution support. Stopping--watchdoes not cancel the run, so if it exits non-zero after returning a run ID, checkair getbefore resubmitting. If the user only wants to submit the run, omit--watchand return the run ID.
Checking on a run
databricks air get <JOB_RUN_ID> -o json -p <profile> # status, config, timing, failure reason
databricks air list -o json -p <profile> # your active runs
databricks air list --all-status -o json -p <profile> # include finished runs
databricks air logs <JOB_RUN_ID> -p <profile> # view logs for an existing run
Read the status from .data.status in the air get -o json envelope. The
terminal statuses are SUCCESS, FAILED, TIMEDOUT, CANCELED,
INTERNAL_ERROR, and SKIPPED. An empty air list means you have no
active runs, not that the run is gone; use --all-status or air get to
see a finished run.
If no waiter is active and you need periodic status snapshots, read
.data.status, sleep between calls, set a hard time limit, and stop on any
terminal status or command error. Because air run --watch waits for the run
to finish, you do not need to poll while it is active.
Monitoring a run
Wait and stream logs. If you want to wait for a new run and receive live
updates, use --watch:
databricks air run --file workload.yaml --watch -p <profile>
For a run that is already going, check whether databricks air logs supports
--follow:
databricks air logs --help 2>&1 | grep -qF -- '--follow'
If that command succeeds, air logs prints the logs available now and exits,
which is the right way to check progress, and --follow streams:
databricks air logs <JOB_RUN_ID> --tail 100 -p <profile> # print the last 100 lines and exit
databricks air logs <JOB_RUN_ID> --follow -p <profile> # stream until the run finishes
# --node N (one node), --tail N (last N lines), --minutes N (last N minutes),
# --retry N (one retry attempt, 0-indexed), --download-to DIR (save all nodes' logs)
If the check fails, the CLI is older: air logs <JOB_RUN_ID> streams an active
run until it finishes, and --follow is rejected. To check progress without
blocking on an older CLI, use databricks air get <JOB_RUN_ID>.
Loss, GPU utilization, and metric history live in MLflow. The CLI streams logs and status, not training metrics. The submit/status response includes the MLflow experiment and run URL; echo it so the user can open the run in the Databricks UI for loss curves, GPU utilization, and system charts.
If a run fails
Get the failure reason, then pull the logs and surface the actual error to the user:
databricks air get <JOB_RUN_ID> -o json -p <profile> # termination_reason
databricks air logs <JOB_RUN_ID> --tail 500 -p <profile> # latest attempt
databricks air logs <JOB_RUN_ID> --retry 0 --tail 500 -p <profile> # first attempt
If the run was retried, the latest attempt's logs may show a different symptom
than the first failure; check --retry 0. On a multi-node run, check other
nodes with --node N when node 0 looks healthy.
torchrun workloads can print hundreds of NCCL INFO lines before the user's
error, so search the logs for Traceback or Error rather than reading them in
order. When creating a UCDataset fails, the traceback ends with the UC error,
such as TABLE_DOES_NOT_EXIST or PERMISSION_DENIED. Under DDP it's a single
long rank 0 failed to build the Delta plan line. In databricks-uc-dataset
0.0.1b1, UC returned no storage location usually means the name is a view,
and set both RANK and WORLD_SIZE or neither means the script wasn't launched
with torchrun. See the UC data checklist.
A TypeError comparing 'float' and 'str' at optimizer or scheduler setup
means a hyperparameter was read without float(); see
Parameters.
Show the failing traceback and let the user decide the fix. Before
resubmitting, set max_retries: 0 so a deterministic error fails once.
Serving a trained model
A model trained with databricks air is served as a
custom model:
log it with MLflow, register it in Unity Catalog, and create a serving endpoint
on CPU or GPU compute.
To serve it with the library versions you tested, without pinning torch or
CUDA for the serving container, use an
express deployment
(Public Preview). The endpoint then runs the environment of the notebook you
registered from. Express deployments require:
- Logging and registering from an AI Runtime notebook to serve on GPU, or from
a serverless notebook to serve on CPU, with
mlflow>=3.12anddatabricks-sdk>=0.102.0. - Registering with
mlflow.register_model(..., env_pack="databricks_model_serving"). Do not useregistered_model_name=onlog_model: it does not acceptenv_packand registers a standard model. - An environment of at most 200 GB.
For LLMs, use
Custom LLM Serving,
which builds on express deployments and starts an OpenAI-compatible server such
as vLLM from the model's entrypoint metadata.
GPU workload_type names map to different GPUs per cloud. For example,
GPU_LARGE is an L40S (48 GB) on AWS and an A100 (80 GB) on Azure. See the GPU
table on the Custom LLM Serving page.
Custom Docker images
Use a custom Docker image instead of environment.version when your workload
needs specific system libraries, CUDA extensions (flash-attn, apex, custom
kernels), or dependencies that do not fit environment.dependencies.
Read docker-images.md for step-by-step guidance on:
- Using Databricks-provided base images, including CUDA 13 images for B300
- Dockerfile patterns
- Pre-build compatibility checklist (CUDA/driver, PyTorch, NCCL, EFA/RDMA)
- Pushing images with
databricks air images pushand running them withenvironment.unity_catalog_image - Troubleshooting runtime errors such as
ssl.SSLErroron FIPS hosts
AI/BI Dashboard Skill
Create Databricks AI/BI dashboards (formerly Lakeview dashboards). A dashboard should be showing something relevant for a human, typically some KPI on the top, and based on the story, some graph (often temporal), and we see "something happens". Follow these guidelines strictly.
Building a dashboard, in order: profile the data (Steps 1-3) → pick widgets from the Widget catalog → read 4-examples.md end to end (a complete dashboard JSON) → read the spec section of each widget you picked (catalog Spec column) and 3-filters.md for filters → build the JSON, test every query, deploy (Step 4, Step 5).
Widget catalog
Wrong version = broken widget! The #1 cause of dashboard errors: use the
vbelow inspec.version. Spec = where the widget's exact JSON is (a section named after it): 1 1-widget-specifications.md, 2 2-advanced-widget-specifications.md, F 3-filters.md, 6 6-custom-visualizations.md. Anchored index: 1- § Widget Index.
Widget (widgetType) | Use for | v | Spec |
|---|---|---|---|
| text (markdown) | titles, section headers, notes; no spec block | – | 1 |
counter | a KPI number, with sparkline and period-over-period delta | 2 | 1 |
table | row-level detail, high-cardinality lists | 2 | 1 |
bar, line | comparisons across categories; trends over time (stacked/grouped, horizontal, annotations) | 3 | 1 |
pie | share of a few categories | 3 | 1 |
area | trends showing volume or stacked composition | 3 | 2 |
scatter | relationship between two measures; bubble = size by a third | 3 | 2 |
combo | bar + line on one chart, metrics with different scales (dual axis) | 1 | 2 |
pivot | cross-tab (dimensions on rows and columns) with conditional cell styling; cohort retention | 3 | 2 |
histogram | frequency distribution of a numeric column | 3 | 2 |
heatmap | "X by Y" matrix, color = aggregate | 3 | 2 |
box | distribution (median, quartiles, outliers) across categories | 1 | 2 |
funnel | stage-by-stage conversion | 1 | 2 |
waterfall | positive/negative deltas to a total (P&L bridge, MoM walk) | 1 | 2 |
sankey | flow between two or more stages | 1 | 2 |
forecast-line | history continued by a forecast band (AI_FORECAST SQL) | 1 | 2 |
gantt | schedules and events with a start and end | 1 | 2 |
symbol-map | lat/lon points (bubble map) | 2 | 1 |
choropleth-map | regions (countries, states) colored by value | 1 | 2 |
path-map | routes and ordered tracks | 1 | 2 |
filter-single-select, filter-multi-select, filter-date-range-picker, filter-date-picker, filter-text-entry | global or page-level filters | 2 | F |
range-slider | numeric range filter | 2 | F |
custom-vega-viz | what standard widgets can't draw (bullet, layered, gauge, radar, sunburst) | 1 | 6 |
There is no cohort widget (use pivot with a color-scale cell style) and no Genie widget (see Linking a Genie Space).
Standard or custom? Use standard widgets by default. Use Vega-Lite when the required visualization cannot be expressed with standard widgets. A different aggregation, color palette, tooltip, reference line, or small multiple usually does not require a custom specification. Case-by-case guide: 6-custom-visualizations.md § Standard widget or custom Vega-Lite?.
For custom charts, confirm the required marks and transforms are supported by Vega-Lite. It does not support every Vega example or chart type. Keep joins and business calculations in dataset SQL; the custom specification controls how the returned data is drawn.
When a custom app fits better: dashboards are for read-only KPIs, charts, and filters over governed tables. For write-back / data entry, UI or interactions beyond the grid, embedded or auth-gated workflows, or a Genie/chat assistant as the main surface, build a Databricks App with
databricks-apps(it brings indatabricks-app-design). Linking an "Ask Genie" space to this dashboard stays here (see Linking a Genie Space below).
Quick Reference
| Task | Command |
|---|---|
| List warehouses | databricks warehouses list |
| List tables | databricks experimental aitools tools query --warehouse WH "SHOW TABLES IN catalog.schema" |
| Get schema | databricks experimental aitools tools discover-schema catalog.schema.table1 catalog.schema.table2 |
| Test query | databricks experimental aitools tools query --warehouse WH "SELECT..." |
| Create dashboard | databricks lakeview create --display-name "X" --warehouse-id "WH" --dataset-catalog CATALOG --dataset-schema SCHEMA --serialized-dashboard "$(cat file.json)" --json '{"parent_path": "/Workspace/Users/<you>/path"}' (flag-only vs JSON-only fields: Step 5) |
| Update dashboard | databricks lakeview update DASHBOARD_ID --dataset-catalog CATALOG --dataset-schema SCHEMA --serialized-dashboard "$(cat file.json)": re-pass both dataset flags (update replaces the dashboard; omitting them nulls the defaults and breaks every bare-table query) |
| Publish / unpublish | databricks lakeview publish DASHBOARD_ID --warehouse-id WH --embed-credentials / databricks lakeview unpublish DASHBOARD_ID: --embed-credentials runs viewers' queries as the publisher so /published works without table access; resets on every publish. New dashboard: pass it unless queries must run as each viewer (RLS / individual permissions). Republish: keep the current mode (Updating). |
| List / get / delete | databricks lakeview list / databricks lakeview get DASHBOARD_ID / databricks lakeview trash DASHBOARD_ID |
| Tag after creation (default; tracks this skill's resources) | databricks workspace-entity-tag-assignments create-tag-assignment dashboards DASHBOARD_ID aidevkit_project --tag-value ai-dev-kit |
--warehouseflag: ifaitools tools queryfails withunknown flag: --warehouseon your CLI version, drop the flag and setDATABRICKS_WAREHOUSE_ID=WHin the environment (picked up automatically).
NEW DASHBOARD CREATION WORKFLOW
You MUST test ALL SQL queries via CLI BEFORE deploying. Follow the overall logic in these steps for new dashboard - Skipping validation causes broken dashboards.
Step 1: Get Warehouse ID if not already known
# List warehouses to find one for SQL execution
databricks warehouses list
Step 2: Discover Table Schemas and existing data pattern
Know the data first: this exploration drives the Step 4 design (which widgets, filters, groupings).
Use discover-schema as the default — one call returns columns, types, sample rows, null counts, and row count. If you only know the schema, list tables first with query "SHOW TABLES IN ...".
databricks experimental aitools tools discover-schema catalog.schema.orders catalog.schema.customers
Then probe with aggregate SQL (cardinality, top values, distributions, trend grain, story confirmation) before designing; fan independent probes out in one multi-query call (--output json, check each statement's state): 7-dashboard-design.md § Profiling the data.
Dashboard queries are different: inside the dashboard JSON use bare table names (✅
FROM trips, ❌FROM samples.nyctaxi.trips); a hardcoded prefix wins over the dataset flags and isn't portable (Dataset architecture).
Step 3: Verify Data Matches Story
Before finalizing, run the SQL Queries you intend to add in each dataset to confirm that they run properly and that the result are valid. Widgets render the query output, so the values must also make sense at a business level.
If values don't match expectations, ensure the query is correct, fix the data if you can, or adjust the story before creating the dashboard.
Step 4: Plan and build the dashboard JSON
- Plan the widgets: pick each one from the Widget catalog, map it to a dataset, and note which filter fields it needs (example plan).
- Read 4-examples.md end to end before writing any JSON. It's a complete working dashboard exercising every construct (dataset measures +
MEASURE(), sparkline counters, forecast-line with annotations, pivot with conditional cells, symbol-map, histogram, range-slider filter, theme). Keep its structure; swap the tables, metrics, palette, and narrative for the user's data and story. - Read the spec section of every widget you picked (catalog Spec column) for its exact JSON and version, and 3-filters.md for filters.
- Check the filters: every dataset you want filtered must contain the filter field (typically group by it). A pre-aggregated table without dates WON'T be date-filtered.
- Build the dashboard JSON as a local working file (intermediate step, not the deliverable), following the skeleton and the implementation guidelines.
Step 5: Deploy
Now deploy the JSON with databricks lakeview create (below). The task isn't complete until it succeeds and returns a dashboard ID; the JSON file is only an intermediate artifact.
Give the user the dashboard link (host: databricks auth env -o json → .env.DATABRICKS_HOST; note the v3, plain /sql/dashboards/ is the legacy path):
- Draft:
https://<host>/sql/dashboardsv3/<DASHBOARD_ID> - Published: append
/published(only valid once you've rundatabricks lakeview publish DASHBOARD_ID)
After deploying, the Quick Reference lakeview commands manage the lifecycle (list, get, update, publish, unpublish, trash, tag).
# parent_path must ALREADY EXIST or create fails "Tree node ... does not exist":
databricks workspace mkdirs /Workspace/Users/me@co.com/dashboards
# Create (mix flags + --json; each field has ONE valid place):
# --dataset-catalog / --dataset-schema: FLAG-ONLY and required. In --json the CLI warns
# "unknown field" and drops them, so no dataset query resolves its catalog.schema.
# parent_path: JSON-ONLY (no flag); without it the dashboard lands at /Users/<you>/<display-name>.
# display_name / warehouse_id / serialized_dashboard: either form; flags read better.
# Capture the id with -o json and don't add 2>&1: create echoes the dashboard back and merged
# stderr breaks jq (the dashboard IS created, but you lose the id and the step looks failed).
DASHBOARD_ID=$(databricks lakeview create \
--display-name "My Dashboard" \
--warehouse-id "abc123def456" \
--dataset-catalog "my_catalog" \
--dataset-schema "my_schema" \
--serialized-dashboard "$(cat dashboard.json)" \
--json '{"parent_path": "/Workspace/Users/me@co.com/dashboards"}' \
-o json | jq -r '.dashboard_id')
# Publish (--embed-credentials: see the CLI table above)
databricks lakeview publish "$DASHBOARD_ID" --warehouse-id "abc123def456" --embed-credentials
UPDATING AN EXISTING DASHBOARD
To change a dashboard that already exists, build the updated JSON with the creation workflow above, then deploy it with update + publish on the same DASHBOARD_ID — never re-run create/import. Commands: Quick Reference (update must re-pass --dataset-catalog / --dataset-schema).
- Don't
create/import to update: it mints a new id + URL and breaks shared links;createis for brand-new dashboards only. updatechanges only the draft:/publishedstays on the last snapshot until youpublishagain, so alwayspublishafterupdate.- Keep the published credential mode: read it first (
databricks lakeview get-published DASHBOARD_ID -o json | jq .embed_credentials); pass--embed-credentialsonly iftrue. Switching from individual permissions to embedded widens what viewers see: only when asked.
JSON Structure (Required Skeleton)
Every dashboard's serialized_dashboard content must follow this exact structure:
{
"datasets": [
{
"name": "ds_x",
"displayName": "Dataset X",
"queryLines": ["SELECT col1, col2 ", "FROM my_table"]
}
],
"pages": [
{
"name": "main",
"displayName": "Main",
"pageType": "PAGE_TYPE_CANVAS",
"layout": [
{"widget": {/* INLINE widget definition */}, "position": {"x":0,"y":0,"width":2,"height":3}}
]
}
]
}
Structural rules (violations cause "failed to parse serialized dashboard"):
queryLines: Array of strings, NOT"query": "string". Elements are joined verbatim with no separator — end each line withor\n(or strip-- comments). A line ending in-- commentwith no newline swallows the next line.- Widgets: INLINE in
layout[].widget, NOT a separate"widgets"array pageType: Required on every page (PAGE_TYPE_CANVASorPAGE_TYPE_GLOBAL_FILTERS)- Query binding:
query.fields[].name==encodings.*.fieldName(rule). Filters bind differently: range sliders exposeMIN/MAXquery fields for onefieldName; parameter-bound date filters usequery.parameters.
Theme & Color (always set this)
Set top-level uiSettings.theme on every dashboard (light/dark canvas, widget, font, selection colors; visualizationColors; widgetHeaderAlignment): the highest-impact polish. Two gotchas: counters take no per-widget color (value.color renders "unsupported widget definition"; the theme fontColor colors their number), and semantic colors go in color.scale.mappings as a bare hex string ({"hex": ...} and themeColorType are silently dropped on charts). Full block, theme keys, palette rules, and starter palettes: 7-dashboard-design.md § Theme & Color.
Linking a Genie Space (Optional)
For an "Ask Genie" button, add uiSettings.genieSpace (isEnabled, overrideId, enablementMode) next to theme. Genie is not a widget (no "widgetType": "assistant"). JSON: 7-dashboard-design.md § Linking a Genie Space.
Design Best Practices
Apply unless the user says otherwise: a global date filter when data has time columns, time-bounded KPIs that support period comparison (MoM, YoY), value formatting by meaning (currency, %, compact K/M/B), and chart type matched to cardinality. Details: 7-dashboard-design.md § Design best practices.
Reference Files
| What are you building? | Reference |
|---|---|
| Start here (read end to end before writing JSON): full working dashboard | 4-examples.md |
| Text, counter, table, bar/line, pie, symbol-map; naming, formatting, field expressions | 1-widget-specifications.md |
| Area, scatter, combo, pivot, histogram, heatmap, box, funnel, waterfall, sankey, forecast-line, gantt, choropleth / path maps | 2-advanced-widget-specifications.md |
| Dashboard with filters (global or page-level) | 3-filters.md |
| Custom Vega-Lite widgets (layered charts, bullet, gauge, radar, sunburst), standard vs custom choice | 6-custom-visualizations.md |
| Data profiling probes, theme & colors, grid layout and sizes, Genie link, data variance | 7-dashboard-design.md |
Dataset-level measures and MEASURE() | 8-dataset-measures.md |
| Debugging a broken dashboard | 5-troubleshooting.md |
Implementation Guidelines
1) DATASET ARCHITECTURE
- Fewer datasets is better: one dataset backing as many widgets as possible. Clicking a chart value (a bar, a slice) filters that dataset, so every widget sharing it re-renders; narrow per-widget datasets break this cross-filtering and force explicit filter widgets. Prefer one wide dataset per domain (orders, cases, customers); split only for a different grain, pre-aggregation, or a parameter the others can't tolerate.
- Two ways to define a dataset: SQL via
queryLines(full control:WITH/JOIN/AI_FORECAST), or the UC asset shorthand{"name": "ds_x", "asset_name": "catalog.schema.table_or_view"}(tables, views, metric views; no SQL). Shapes: 8-dataset-measures.md § Defining a dataset. - Exactly ONE valid SQL query per dataset when using
queryLines(no multiple queries separated by;) - Queries must use bare table names only — no catalog, no schema prefix. Example:
FROM orders, neverFROM gold.ordersorFROM main.gold.orders. The catalog and schema come from the--dataset-catalogand--dataset-schemaflags at creation time. These flags only fill in missing parts — they do NOT override any catalog/schema written in the query. - SELECT must include all dimensions needed by widgets and all derived columns via
ASaliases - Put ALL business logic (CASE/WHEN, COALESCE, ratios) into the dataset SELECT with explicit aliases
- Add ORDER BY when visualization depends on data order:
- Time series:
ORDER BY datefor chronological display - Rankings/Top-N:
ORDER BY metric DESC LIMIT 10for "Top 10" charts - Categorical charts:
ORDER BY metric DESCto show largest values first
- Time series:
Dataset-level measures + MEASURE()
Declare reusable measures in a dataset's columns (or rely on a metric view's YAML measures, without redeclaring them) and reference them from any widget as MEASURE(`Name`) (counter, table, bar, line, pie, pivot; mixes with inline aggregations). Shapes and examples: 8-dataset-measures.md.
2) WIDGET FIELD EXPRESSIONS
CRITICAL: Field Name Matching Rule For chart, table, counter, and custom-viz widgets, each selected
nameinquery.fieldsMUST exactly match itsfieldNameinencodings. If they don't match, the widget shows "no selected fields to visualize" error!
For custom-vega-viz the binding has extra layers (spec.encodings.fields, spec.data.queryName, Vega-Lite data.name: "databricks_query", spec as a JSON string in spec.jsonSpec.spec): read 6-custom-visualizations.md § Widget Contract before generating one.
Correct vs mismatched example: 5-troubleshooting.md § no selected fields.
Allowed expressions in widget queries (you CANNOT use CAST or other SQL in expressions):
{"name": "(sum|avg|count|countdistinct|min|max)(col)", "expression": "(SUM|AVG|COUNT|COUNT(DISTINCT)|MIN|MAX)(`col`)"}
{"name": "(daily|weekly|monthly)(date)", "expression": "DATE_TRUNC(\"(DAY|WEEK|MONTH)\", `date`)"}
{"name": "field", "expression": "`field`"}
If you need conditional logic or multi-field formulas, compute a derived column in the dataset SQL first.
Some widgets support additional expressions: Gantt can use FIRST(...) for status fields, and geometry map bindings require ST_ASGEOJSON(...). See the corresponding widget reference for these shapes.
3) SPARK SQL PATTERNS
- Date math:
date_sub(current_date(), N)for days,add_months(current_date(), -N)for months - Date truncation:
DATE_TRUNC('DAY'|'WEEK'|'MONTH'|'QUARTER'|'YEAR', column) - AVOID
INTERVALsyntax - use functions instead
4) LAYOUT (12-Column Grid, NO GAPS)
Every page must include "layoutVersion": "GRID_V1" alongside pageType. Positions are {"x", "y", "width", "height"}; by default each row fills width 12 exactly (no gaps). Counters height 3-4 (never 2), charts 5-6, text headers 12x1 with title and subtitle as separate widgets. Column subdivision by audience, the size table, a standard page structure, and staggered layouts: 7-dashboard-design.md § Layout.
5) CARDINALITY & READABILITY (CRITICAL)
Dashboard readability depends on limiting distinct values:
| Dimension Type | Max Values | Examples |
|---|---|---|
| Chart color/groups | 3-8 | 4 regions, 5 product lines, 3 tiers |
| Filters | 4-15 | 8 countries, 5 channels |
| High cardinality | Table only | customer_id, order_id, SKU |
Before creating any chart with color/grouping:
- Check column cardinality via discover-schema or a COUNT DISTINCT query
- If >10 distinct values, aggregate to higher level OR use TOP-N + "Other" bucket
- For high-cardinality dimensions, use a table widget instead of a chart
Before a trend chart, check variance: a small MAX - MIN range (e.g. 83-89% on 0-100) looks flat; prefer a KPI with delta or a table (details).
6) QUALITY CHECKLIST
Before deploying, verify:
- All widget names use only alphanumeric + hyphens + underscores
- Every page has
"layoutVersion": "GRID_V1" - All rows sum to width=12 with no gaps
- KPIs use height 3-4, charts use height 5-6
- Chart dimensions have reasonable cardinality (≤8 for colors/groups)
- All widget query expressions reference existing dataset columns or aliases
- Visualization field bindings match exactly (charts, tables, counters, custom viz:
query.fields[].name== encodingfieldName, e.g. both"sum(spend)"); validate filters against their own contracts. - Counter datasets: use
disaggregated: truefor 1-row datasets,disaggregated: falsewith aggregation for multi-row - Percent values must be 0-1 for
number-percentformat (0.865 displays as "86.5%", don't forget to set the format). If data is 0-100, either divide by 100 in SQL or usenumberformat instead. - SQL uses Spark syntax (date_sub, not INTERVAL)
- All SQL queries tested via CLI and return expected data
- Every dataset you want filtered MUST contain the filter field — filters only affect datasets with that column in their query
abbreviation: "compact"needsdecimalPlaces— without it the value isn't rounded (renders$9.756278496M, not$9.76M)
Related Skills
- databricks-apps - a custom-code interactive app (write-back, bespoke UI, in-app chat / Genie) instead of a managed dashboard
- databricks-unity-catalog - for querying the underlying data and system tables
- databricks-pipelines - for building the data pipelines that feed dashboards
- databricks-jobs - for scheduling dashboard data refreshes
Data App Design
Make Databricks data + AI apps that communicate clearly and compile to real AppKit code. This skill merges two bodies of knowledge and binds them to implementation:
- Composition — what to show, how much to abstract, how to lay it out →
references/dashboard-patterns.md - Notation — make comparable things look comparable; honest scales; scenario marks →
references/ibcs-notation.md - Implementation — the exact AppKit components, hooks, and tokens to use →
references/appkit-cheatsheet.md
For package scripts and project-local CLI tools, use the parent skill's package-manager selection and command placeholders.
Design advice that doesn't name a real component is incomplete. Always end at a component plan.
When to use / when NOT
- USE for: the data screens of a custom-code Databricks App (AppKit/React) — overview/KPI pages, reports, metric/ontology pages, variance analysis, charts, tables, and Genie/NL data surfaces — design or critique.
- Do NOT use for: authoring managed AI/BI (Lakeview) dashboards (→
databricks-aibi-dashboards), generic frontend (forms, auth, settings, marketing), or scaffolding/build/deploy (→databricks-apps). A plain "create a dashboard" / "build a dashboard" request (no app / AppKit / React / custom-code signal) means a managed AI/BI (Lakeview) dashboard → usedatabricks-aibi-dashboards, not this skill. If a request is "add a form", "deploy this", or "build a Lakeview / AI-BI dashboard", this skill should not fire. - Relationship:
databricks-appsbuilds/runs the app; this skill decides what the data screens should look like and which primitives realize them.
Workflow
- Frame — audience, the decision/question, refresh cadence, device, primary task. One sentence.
- Genre — pick the closest from
references/dashboard-patterns.md(static / analytic / magazine / infographic / repository / embedded mini). State it. - Compose — choose content + composition patterns (data abstraction, meta-info, layout, interaction, color). Make the tradeoff explicit: what's summarized, hidden, paginated, or made interactive — and why.
- Apply notation — run the relevant
references/ibcs-notation.mdrules: message-in-title, scenario marks (actual/PY/plan/forecast), honest scales, semantic color. On any chart-vocabulary conflict, IBCS wins (see the conflict note in that file). - Bind to components — map every element to a primitive that's actually exported from
@databricks/appkit/@databricks/appkit-ui(seereferences/appkit-cheatsheet.md); never cite a component AppKit doesn't ship. There's no prebuilt KPI/trend/distribution card — compose those from primitives, following the notation rules. UsecolorPalette+ semantic tokens, never hardcoded hex. Bind data withuseAnalyticsQuery/queryKey+sql.*params. - Cover the states — every data view must handle loading / empty / error / partial (see checklist).
- Review — run the checklists in both reference files; lead critiques with the highest-impact comprehension or integrity issue, citing the affected component/file.
Required states & data realism (non-negotiable for data apps)
- Loading →
Skeleton; Empty →Emptywith a useful next action; Error → inline message, never a blank panel; Partial/stale → show what you have + a freshness note. - Every KPI shows unit + period + comparison + freshness/source (mirror the metric definition; don't show a number with no provenance).
- Large tables → server-side pagination/sort/filter, not client-side over a huge result set.
- Long-running queries → optimistic loading + timeout/error UX.
AI / Genie surfaces (the "AI" half)
Gate: this section applies only if the app has a Genie / chat / natural-language / "ask your data" surface. For a pure dashboard / KPI / report app with no conversational input, skip this section and references/genie-ai-trust.md entirely. When it does apply, implement ALL five (code in references/genie-ai-trust.md):
A Genie/chat/NL answer is only trustworthy if the user can see how it was produced and who it ran as. "Use GenieChat + a spinner" is NOT enough — for ANY Genie/chat surface, ship all five (copy the exact snippets from the reference):
- Identity — a
/api/whoamiroute (realx-forwarded-email/x-forwarded-userheaders) + the signed-in user in aBadge. Claim OBO only ifuser_api_scopes: [dashboards.genie]is wired; otherwise disclose the query runs as the app's service principal. - Generated SQL — render
attachments[].queryin an inspectable "Generated SQL"Card; never hide how the answer was computed. - Streaming/status — reflect
useGenieChat().status(streaming/error), never a frozen spinner. - Disclaimer — a persistent "AI-generated — verify" note per answer.
- Governance + states —
genie()space config + a truthful execution-identity note (OBO when user-scoped, else service principal) + empty/error/ambiguous handling (Empty,Alert).
Output formats
Design proposal:
## Direction
[Genre, audience, primary task, design intent.]
## Pattern & notation choices
- Composition: [data info, meta info, layout, interaction, color]
- Notation: [message, scenario marks, scales, semantic color]
## Component plan ← the part that makes it buildable
- [element] → [AppKit component] (queryKey/props), [token/palette], states handled
## Tradeoffs & risks
[What's summarized/hidden/paginated/interactive; overload, scale, a11y, maintenance risks.]
Critique: lead with the top comprehension/integrity issue, cite the component/file, then list findings by impact, each with the concrete fix (which component/token/state to change).
Anti-patterns
- Producing a design memo with no component plan.
- "Use semantic color" without naming the token/palette.
- Naming a component AppKit doesn't export (e.g. a prebuilt
KpiCard) — compose composites from published primitives instead. - Adding interaction, pages, or density the task doesn't need (over-engineering a mock-first app).
- Forgetting loading/empty/error states, or KPIs with no freshness/source.
Databricks Applications — Python backends
First, confirm this skill is the right one. The default for new Databricks Apps is databricks-apps (AppKit — Node.js + TypeScript + React SDK). Load that skill first unless the user explicitly asks for a Python backend, is extending an existing Python app, or the team is Python-only. Everything below is the Python-backend alternative.
Critical Rules for Python apps (always follow)
- MUST confirm framework choice or use Python Framework Selection below
- MUST use SDK
Config()for authentication (never hardcode tokens) - MUST use
app.yamlvalueFromfor resources (never hardcode resource IDs) - MUST use
dash-bootstrap-componentsfor Dash app layout and styling - MUST use
@st.cache_resourcefor Streamlit database connections - MUST deploy Flask with Gunicorn, FastAPI with uvicorn (not dev servers)
Required Steps for Python apps
Copy this checklist and verify each item:
- [ ] Framework selected
- [ ] Auth strategy decided: app auth, user auth, or both
- [ ] App resources identified (SQL warehouse, Lakebase, serving endpoint, etc.)
- [ ] Backend data strategy decided (SQL warehouse, Lakebase, or SDK)
- [ ] Deployment method: CLI or DABs
Python Framework Selection
| Framework | Best For | app.yaml Command |
|---|---|---|
| FastAPI (default) | Any Python backend by default — async APIs, auto-generated OpenAPI docs, JSON-serving apps | ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"] |
| Flask | Custom REST APIs, lightweight apps, webhooks | ["gunicorn", "app:app", "-w", "4", "-b", "0.0.0.0:8000"] |
| Dash | Production dashboards, BI tools, complex interactivity | ["python", "app.py"] |
| Streamlit | Rapid prototyping, data science apps, internal tools where the UI is a series of Python widgets | ["streamlit", "run", "app.py"] |
| Gradio | ML demos, model interfaces, chat UIs | ["python", "app.py"] |
| Reflex | Full-stack Python apps without JavaScript | ["reflex", "run", "--env", "prod"] |
Default: FastAPI. Reach for FastAPI unless the user explicitly asks for Streamlit-style widget prototyping (Streamlit), a heavy dashboard grid (Dash), or a Gradio-style ML demo. FastAPI pairs naturally with a JS/HTML frontend or a JSON-consuming caller — the same posture databricks-apps uses on the Node side.
Quick Reference
| Concept | Details |
|---|---|
| Runtime | Python 3.11, Ubuntu 22.04, 2 vCPU, 6 GB RAM |
| Pre-installed | Dash 2.18.1, Streamlit 1.38.0, Gradio 4.44.0, Flask 3.0.3, FastAPI 0.115.0 |
| Auth (app) | Service principal via Config() — auto-injected DATABRICKS_CLIENT_ID/DATABRICKS_CLIENT_SECRET |
| Auth (user) | x-forwarded-access-token header — see references/1-authorization.md |
| Resources | valueFrom in app.yaml — see references/2-app-resources.md |
| SDK / Foundation Models / Model Serving | Use the databricks-python-sdk skill — same WorkspaceClient and OpenAI-compatible foundation-model patterns work inside a Databricks App. For UC model services (system.ai.*), see the model-service note in references/2-app-resources.md and the databricks-unity-gateway skill |
| AI Search (Vector Search) | Use the databricks-vector-search skill: AISearchClient(workspace_url=..., service_principal_client_id=..., service_principal_client_secret=...) from DATABRICKS_HOST (prefix https:// if missing), DATABRICKS_CLIENT_ID, and DATABRICKS_CLIENT_SECRET. Create the client and call get_index() once at startup, then reuse the index object across requests. Add databricks-ai-search>=0.83 to the app's dependencies (e.g. requirements.txt). WorkspaceClient queries are capped at a few tens of QPS |
| Docs | https://docs.databricks.com/dev-tools/databricks-apps/ |
Detailed Guides
Authorization: Use references/1-authorization.md when configuring app or user authorization — covers service principal auth, on-behalf-of user tokens, OAuth scopes, and per-framework code examples. (Keywords: OAuth, service principal, user auth, on-behalf-of, access token, scopes)
App resources: Use references/2-app-resources.md when connecting your app to Databricks resources — covers SQL warehouses, Lakebase, model serving, secrets, volumes, and the valueFrom pattern. (Keywords: resources, valueFrom, SQL warehouse, model serving, secrets, volumes, connections)
Frameworks: See references/3-frameworks.md for Databricks-specific patterns per framework — FastAPI (default), Flask, Dash, Streamlit, Gradio, Reflex — with auth integration and deployment commands. (Keywords: FastAPI, Flask, Dash, Streamlit, Gradio, Reflex, framework selection)
Deployment: Use references/4-deployment.md when deploying your app — covers Databricks CLI, Declarative Automation Bundles (DABs), app.yaml configuration, and post-deployment verification. (Keywords: deploy, CLI, DABs, asset bundles, app.yaml, logs)
Lakebase: Use references/5-lakebase.md when using Lakebase (PostgreSQL) as your app's data layer — covers auto-injected env vars, psycopg2/asyncpg patterns, and when to choose Lakebase vs SQL warehouse. (Keywords: Lakebase, PostgreSQL, psycopg2, asyncpg, transactional, PGHOST)
CLI commands: Use references/6-cli-approach.md for managing app lifecycle via CLI — covers creating, deploying, monitoring, and deleting apps. (Keywords: CLI, create app, deploy app, app logs)
Foundation Models / SDK / Model Serving: Use the databricks-python-sdk skill for the OpenAI-compatible foundation-model client, WorkspaceClient calls, and model-serving invocation — the same patterns apply inside a Databricks App. For AI Search (Vector Search) queries, use the databricks-vector-search skill instead (AISearchClient with the app's service principal; see Production and High QPS). The examples in this skill's examples/ folder (fm-minimal-chat.py, fm-parallel-calls.py, fm-structured-outputs.py, llm_config.py) show the App-side wiring only. For UC model services (system.ai.*, called on /ai-gateway/mlflow/v1), see the model-service note in references/2-app-resources.md and the databricks-unity-gateway skill.
Workflow
-
Determine the task type:
New app from scratch? → Load databricks-apps first (AppKit / Node). Only stay in this skill if the user explicitly asks for a Python backend. Python-backend confirmed? → Python Framework Selection — default to FastAPI. Setting up authorization? → Read references/1-authorization.md Connecting to data/resources? → Read references/2-app-resources.md Using Lakebase (PostgreSQL)? → Read references/5-lakebase.md Deploying to Databricks? → Read references/4-deployment.md Using CLI for app lifecycle? → Read references/6-cli-approach.md Calling foundation model / LLM APIs or model-serving endpoints? → Load the databricks-python-sdk skill. This skill's
examples/folder shows only the App-side wiring on top of those SDK patterns. Querying AI Search (Vector Search)? → Load the databricks-vector-search skill. -
Follow the instructions in the relevant guide.
Core Architecture
All Python Databricks apps follow this pattern:
app-directory/
├── app.py # Main application (or framework-specific name)
├── models.py # Pydantic data models
├── backend.py # Data access layer
├── requirements.txt # Additional Python dependencies
├── app.yaml # Databricks Apps configuration
└── README.md
Backend Toggle Pattern
import os
from databricks.sdk.core import Config
USE_MOCK = os.getenv("USE_MOCK_BACKEND", "true").lower() == "true"
if USE_MOCK:
from backend_mock import MockBackend as Backend
else:
from backend_real import RealBackend as Backend
backend = Backend()
SQL Warehouse Connection (shared across all frameworks)
from databricks.sdk.core import Config
from databricks import sql
cfg = Config() # Auto-detects credentials from environment
conn = sql.connect(
server_hostname=cfg.host,
http_path=f"/sql/1.0/warehouses/{os.getenv('DATABRICKS_WAREHOUSE_ID')}",
credentials_provider=lambda: cfg.authenticate,
)
Pydantic Models
from pydantic import BaseModel, Field
from datetime import datetime
from enum import Enum
class Status(str, Enum):
ACTIVE = "active"
PENDING = "pending"
class EntityOut(BaseModel):
id: str
name: str
status: Status
created_at: datetime
class EntityIn(BaseModel):
name: str = Field(..., min_length=1)
status: Status = Status.PENDING
Common Issues
| Issue | Solution |
|---|---|
| Connection exhausted | Use @st.cache_resource (Streamlit) or connection pooling |
| Auth token not found | Check x-forwarded-access-token header — only available when deployed, not locally |
| App won't start | Check app.yaml command matches framework; check databricks apps logs <name> |
| Resource not accessible | Add resource via UI, verify SP has permissions, use valueFrom in app.yaml |
| Import error on deploy | Add missing packages to requirements.txt (pre-installed packages don't need listing) |
| Lakebase app crashes on start | psycopg2/asyncpg are NOT pre-installed — MUST add to requirements.txt |
| Port conflict | Apps must bind to DATABRICKS_APP_PORT env var (defaults to 8000). Never use 8080. Streamlit is auto-configured; for others, read the env var in code or use 8000 in app.yaml command |
| Streamlit: set_page_config error | st.set_page_config() must be the first Streamlit command |
| Dash: unstyled layout | Add dash-bootstrap-components; use dbc.themes.BOOTSTRAP |
| Slow queries | Use Lakebase for transactional/low-latency; SQL warehouse for analytical queries |
Platform Constraints
| Constraint | Details |
|---|---|
| Runtime | Python 3.11, Ubuntu 22.04 LTS |
| Compute | 2 vCPUs, 6 GB memory (default) |
| Pre-installed frameworks | Dash, Streamlit, Gradio, Flask, FastAPI, Shiny |
| Custom packages | Add to requirements.txt in app root |
| Network | Apps can reach Databricks APIs; external access depends on workspace config |
| User auth | GA, on by default — no admin enablement required; declare user_api_scopes to add scopes |
Official Documentation
- Databricks Apps Overview — main docs hub
- Authorization — app auth and user auth
- Resources — SQL warehouse, Lakebase, serving, secrets
- app.yaml Reference — command and env config
- System Environment — pre-installed packages, runtime details
Related Skills
- databricks-apps — the default for new Databricks Apps (AppKit / Node / TypeScript + React); load it first unless a Python backend is explicitly required
- databricks-python-sdk —
WorkspaceClient, OpenAI-compatible foundation-model client, model-serving invocation; the same patterns work inside a Databricks App - databricks-vector-search — AI Search (Vector Search) indexes and high-QPS queries with
AISearchClient - databricks-lakebase — persistent PostgreSQL state (autoscaling managed PG with branching)
- databricks-model-serving — endpoint lifecycle for ML models an App calls
- databricks-dabs — deploying apps via DABs
Databricks Apps Development
FIRST: Use the parent databricks-core skill for CLI basics, authentication, and profile selection.
Package manager — resolve this before any install or command below:
Databricks Apps officially supports npm and pnpm only.
- Detect the project's manager, in priority order: project instructions →
package.json'spackageManagerfield → lockfile (pnpm-lock.yaml→ pnpm,package-lock.json→ npm). - If signals conflict, or if an unsupported manager (yarn, bun, etc.) is detected, stop and ask the user which manager to use before proceeding.
- Only if none is established, default to pnpm.
Throughout this skill and its guides, substitute what you resolved:
<pm>→ that manager, for installs and package scripts (<pm> install,<pm> run typegen).
<pm> is a documentation placeholder, not a shell variable — expand it, don't type it.
Is this even a Databricks App? Don't assume an app is the only way to show data. For a simple "dashboard with a few charts" and no app-specific need, the managed AI/BI (Lakeview) dashboard is the simpler path → use the databricks-aibi-dashboards skill, not this one. Reach for a custom Databricks App only when the user needs something AI/BI can't give them — bespoke interactivity/components, write-back, embedded or auth-gated workflows, a Genie/chat surface inside the app — or explicitly asks for an app. If it's genuinely ambiguous, surface both options (managed AI/BI dashboard vs custom app) and let the user choose instead of defaulting to an app. Once an app is the right call, it's fine for this skill to favor an app-based dashboard, and databricks-app-design covers building it well.
For data UI design (required for any data-displaying app): once you've confirmed the user wants a custom-code app (not a managed AI/BI dashboard — see above), if the app shows ANY data — a KPI/overview page, report, chart, table, query results, OR a conversational / chat / Genie natural-language assistant — you MUST use the databricks-app-design skill (alongside this one) to decide layout, charts, KPIs, semantic color, required states, and AI-result trust, and map them to AppKit components. This includes chat/Genie apps, not just static data views — if in doubt, use it.
Build apps that deploy to Databricks Apps platform.
Required Reading by Phase
| Phase | READ BEFORE proceeding |
|---|---|
| Scaffolding | ⚠️ STOP — review the State Storage Guidance and complete the Data Access Decision Gate below before scaffolding. Parent databricks-core skill (auth, warehouse discovery); then run databricks apps manifest + databricks apps init with --features and --set (see AppKit section below) |
| Writing SQL queries | SQL Queries Guide |
| Writing UI components | Frontend Guide |
Using useAnalyticsQuery | AppKit SDK |
| Querying a governed UC Metric View | Metric Views Guide |
| Adding API endpoints | Custom Endpoints Guide |
| Using Lakebase (OLTP database) | Lakebase Guide |
| Adding Genie chat / Genie-powered apps | Genie Guide — follow the Genie agent workflow below |
| Using Model Serving (ML inference) | Model Serving Guide |
| Generating or editing images (text-to-image, image-to-image) | Image Generation Guide |
| Hosting an AI agent (tool-using chatbot, beta) | Agents Guide — import from @databricks/appkit/beta |
| Typed data contracts (proto-first design) | Proto-First Guide and Plugin Contracts |
| Managing files in UC Volumes | Files Guide |
| Triggering / monitoring Lakeflow Jobs from the app | Jobs Guide |
| Platform rules (permissions, deployment, limits) | Platform Guide — READ for ALL apps including AppKit |
| Non-AppKit app (Streamlit, FastAPI, Flask, Gradio, Next.js, etc.) | Other Frameworks |
Generic Guidelines
- App name: Platform App resource names allow up to 30 lowercase letters, numbers, and hyphens (no underscores). For AppKit projects created with
databricks apps init, keep the base name to 26 characters because its development workflow adds adev-prefix. DABs does not add that prefix. - Validation:
databricks apps validate --profile <PROFILE>before deploying. - Smoke tests (AppKit only): ALWAYS update
tests/smoke.spec.tsselectors BEFORE running validation. Default template checks for "Minimal Databricks App" heading and "hello world" text — these WILL fail in your custom app. See testing guide (selector APIs, the 1 MB result-size cap). - AppKit version: never override the
@databricks/appkitor@databricks/appkit-uiversion inpackage.json—databricks apps initsets the correct version. Do not install a different version of either package unless explicitly asked by the user. If you need a different version, re-scaffold withdatabricks apps init --version <version>. - Authentication: covered by parent
databricks-coreskill. - AppKit API surface: before writing code that calls AppKit APIs (
createApp, plugin shapes,useAnalyticsQuery,useMetricView, etc.), runnpx @databricks/appkit docs <section>and use the actual signature. Training data has stale shapes; a single invented signature failstsc --noEmitduring validate. The docs ship with the installed AppKit and are the authoritative source. - TypeScript casts: never use
as unknown as <T>double-assertions —appkit lintenforcesno-double-type-assertionand one violation fails the entire validate step. Instead: narrow with Zod (z.infer<typeof schema>), use a runtime type guard, or write a typed mapper function. If a query result needs reshaping, type the row schema via queryKey types rather than casting.
Project Structure (after databricks apps init --features analytics)
client/src/App.tsx— main React component (start here)config/queries/*.sql— SQL query files (queryKey = filename without .sql)config/metric-views/definitions.json— governed UC Metric View bindings (optional; see Metric Views)server/server.ts— backend entry (onPluginsReady+ Express routes)tests/smoke.spec.ts— smoke test (⚠️ MUST UPDATE selectors for your app)client/src/appKitTypes.d.ts— auto-generated types (<pm> run typegen)
Project Structure (after databricks apps init --features lakebase)
server/server.ts— backend with Lakebase pool + Express routesclient/src/App.tsx— React frontendapp.yaml— manifest withdatabaseresource declarationpackage.json— includes@databricks/lakebasedependency- Note: No
config/queries/— Lakebase apps useappkit.lakebase.query()in Express routes, not SQL files
Data Discovery
Before writing any SQL, use the parent databricks-core skill for data exploration — search information_schema by keyword, then batch discover-schema for the tables you need. Do NOT skip this step.
State Storage Guidance (evaluate BEFORE the Decision Gate):
If the user's app description involves storing or persisting data — forms, CRUD operations, user submissions, orders, todos, or other user-generated content — the app likely needs a Lakebase database.
- Ask the user whether the app needs persistent storage (Lakebase) before scaffolding. Do not silently add Lakebase.
- If confirmed, ask whether to reuse an existing Lakebase project or create a new one (same as the Genie flow below) — never create one silently. Use the
databricks-lakebaseskill to obtain the branch and database resource names:- Reusing: list with
databricks postgres list-projects, thenlist-branches/list-databases, and let the user pick the project, branch, and database; confirm which schema the app will own (a fresh/dedicated schema avoids the service-principal ownership conflict). - Creating: create a new project (it auto-provisions a
productionbranch +databricks_postgresdatabase).
- Reusing: list with
- Scaffold with
--features lakebaseand pass--set lakebase.postgres.branch=<BRANCH_NAME> --set lakebase.postgres.database=<DATABASE_NAME>. - If the app also reads from Unity Catalog tables, proceed to the Data Access Decision Gate below to determine whether to add
--features analyticsor use Lakebase synced tables.
Do NOT add Lakebase to analytics, dashboard, or visualization apps unless the user explicitly requests persistent write-back storage. Read-only data display, filters, and preferences do not require a database.
Development Workflow (FOLLOW THIS ORDER)
Data Access Decision Gate (REQUIRED before scaffolding):
If the app reads from Unity Catalog / lakehouse tables, you MUST show the comparison below to the user and ask them to choose. Do not skip this. Do not choose for them.
| (A) Lakebase synced tables | (B) Analytics | |
|---|---|---|
| Speed | Sub-second responses | Takes a few seconds |
| Best for | Full-text search, typeahead, autocomplete, real-time lookups, operational apps | Dashboards, charts, aggregations, KPIs, filtered queries, browsing |
| How it works | Data synced from Delta into Lakebase Postgres | Queries run on SQL warehouse at read time |
After showing the table, add a brief recommendation. Default to recommending Analytics (B) for most read-only apps — dashboards, charts, filtered queries, browsing, and aggregations. Recommend Lakebase synced tables (A) only when the app needs sub-second latency for full-text search, typeahead/autocomplete, real-time lookups by ID, or operational data serving. Note: "search" or "filter" in a prompt usually means SQL WHERE clauses (Analytics), not full-text search (Lakebase). Always let the user make the final call.
After the user chooses:
- (A) Lakebase synced tables → scaffold with
--features lakebase. See Lakebase Guide for full workflow. - (B) Analytics → scaffold with
--features analytics. Within Analytics, if the data is a governed UC Metric View (pre-defined semantic measures/dimensions), query it via the metric-view path — see Metric Views — otherwise writeconfig/queries/SQL. - Both → scaffold with
--features analytics,lakebaseif the app needs both patterns. - If the app does NOT read Unity Catalog data (pure CRUD, Genie, Model Serving), skip this gate and scaffold with the appropriate
--featuresflag.
Analytics apps (--features analytics):
- Create SQL files in
config/queries/ - Run
<pm> run typegen— verify all queries show ✓ - Read
client/src/appKitTypes.d.tsto see generated types - THEN write
App.tsxusing the generated types - Update
tests/smoke.spec.tsselectors - Run
databricks apps validate --profile <PROFILE>
DO NOT write UI code before running typegen — types won't exist and you'll waste time on compilation errors.
Lakebase apps (--features lakebase): No SQL files or typegen. See Lakebase Guide for the onPluginsReady pattern: initialize schema at startup, register Express routes in server/server.ts, then build the React frontend.
When to Use What
After completing the decision gate above, use this routing table:
- Read analytics data → display in chart/table: Use visualization components with
queryKeyprop - Read analytics data → custom display (KPIs, cards): Use
useAnalyticsQueryhook - Read analytics data → need computation before display: Still use
useAnalyticsQuery, transform client-side - Read a governed UC Metric View (semantic layer): Use the
useMetricViewhook +config/metric-views/definitions.json— see Metric Views - Read lakehouse data at low latency (lookups, search, catalogs): Use Lakebase synced tables — see Lakebase Guide
- Read/write persistent data (users, orders, CRUD state): Use Lakebase via Express routes in
onPluginsReady— see Lakebase Guide - Natural language query interface over tables (Genie): Use
genie()plugin — see Genie Guide - Call ML model endpoint: Use
serving()plugin — see Model Serving Guide - Call a foundation model / UC model service (
system.ai.*): Declare auc_securable(MODEL_SERVICE) resource and call the service name on/ai-gateway/mlflow/v1— see Unity Catalog model services and thedatabricks-unity-gatewayskill (notserving(), which takes serving-endpoint names) - Generate or edit images from a prompt: streaming Express route (or an
agentstool for chat UIs) + nativefetchto an image-capable Foundation Model — see Image Generation Guide - AI agent that calls tools (SQL, files, Genie, MCP), delegates to sub-agents, or streams a multi-turn chat with human-in-the-loop approval: Use the
agents()plugin (import from@databricks/appkit/beta, beta) — see Agents Guide - Trigger or monitor a Lakeflow Job from the app: Use the
jobs()plugin — see Jobs Guide - ⚠️ NEVER add custom endpoints to run SELECT queries against the warehouse — always use SQL files in
config/queries/ - ⚠️ NEVER use
useAnalyticsQueryfor Lakebase data — it queries the SQL warehouse only
Frameworks
AppKit (Recommended)
TypeScript/React framework with type-safe SQL queries and built-in components.
Official Documentation — the source of truth for all API details. Use the project-local appkit CLI from the installed @databricks/appkit package:
npx @databricks/appkit docs # ← ALWAYS start here to see available pages
npx @databricks/appkit docs <query> # view a section by name or doc path
npx @databricks/appkit docs --full # full index with all API entries
npx @databricks/appkit docs "appkit-ui API reference" # example: section by name
npx @databricks/appkit docs ./docs/plugins/analytics.md # example: specific doc file
DO NOT guess doc paths. Run without args first, pick from the index. The <query> argument accepts both section names (from the index) and file paths. Docs are the authority on component props, hook signatures, and server APIs — skill files only cover anti-patterns and gotchas.
App Manifest and Scaffolding
Agent workflow for scaffolding: get the manifest first, then build the init command.
-
Get the manifest (JSON schema describing plugins and their resources):
databricks apps manifest --profile <PROFILE> # See plugins available in a specific AppKit version: databricks apps manifest --version <VERSION> --profile <PROFILE> # Custom template: databricks apps manifest --template <GIT_URL> --profile <PROFILE>The output defines:
- Plugins: each has a key (plugin ID for
--features), plusrequiredByTemplate, andresources. - requiredByTemplate: If true, that plugin is mandatory for this template — do not add it to
--features(it is included automatically); you must still supply all of its required resources via--set. If false or absent, the plugin is optional — add it to--featuresonly when the user's prompt indicates they want that capability (e.g. analytics/SQL), and then supply its required resources via--set. - Resources: Each plugin has
resources.requiredandresources.optional(arrays). Each item hasresourceKeyandfields(object: field name → description/env). Use--set <plugin>.<resourceKey>.<field>=<value>for each required resource field of every plugin you include.
- Plugins: each has a key (plugin ID for
-
Scaffold (use the Databricks CLI only):
databricks apps init --name <NAME> --features <plugin1>,<plugin2> \ --set <plugin1>.<resourceKey>.<field>=<value> \ --set <plugin2>.<resourceKey>.<field>=<value> \ --description "<DESC>" --run none --profile <PROFILE> # --run none: skip auto-run after scaffolding (review code first) # With custom template: databricks apps init --template <GIT_URL> --name <NAME> --features ... --set ... --profile <PROFILE>Optionally use
--version <VERSION>to target a specific AppKit version.- Required:
--name,--profile. Name: ≤26 chars, lowercase letters/numbers/hyphens only. Use--featuresonly for optional plugins the user wants (plugins withrequiredByTemplate: falseor absent); mandatory plugins must not be listed in--features. - Resources: Pass
--setfor every required resource (each field inresources.required) for (1) all plugins withrequiredByTemplate: true, and (2) any optional plugins you added to--features. Add--setforresources.optionalonly when the user requests them. - Discovery: Use the parent
databricks-coreskill to resolve IDs (e.g. warehouse:databricks warehouses list --profile <PROFILE>ordatabricks experimental aitools tools get-default-warehouse --profile <PROFILE>).
- Required:
DO NOT guess plugin names, resource keys, or property names — always derive them from databricks apps manifest output. Example: if the manifest shows plugin analytics with a required resource resourceKey: "sql-warehouse" and fields: { "id": ... }, include --set analytics.sql-warehouse.id=<ID>.
Execution identity (auth mode): --auth-mode obo|sp (app default) and per-resource --set <plugin>.<resourceKey>.authMode=obo|sp|both need databricks CLI >= v1.20.0; omit them on older CLIs (everything binds to the service principal). Default to SP; choose OBO only when the app must enforce the viewing user's own permissions (row-level or per-user data access). Flag rules, errors, and what each mode emits: Platform Guide.
Scaffolding Rules Protocol: if databricks apps manifest emits scaffolding.rules (template-level and per plugin, as must / should / never), they override this skill. Run Before init rules before databricks apps init and After init rules after it; if a plugin must contradicts a template never on the same target, STOP and ask the user. Gather, precedence, and reporting steps: AppKit Overview § Scaffolding Rules Protocol.
READ AppKit Overview for project structure, workflow, and pre-implementation checklist.
Genie Agent Workflow: for a Genie-powered app, ask which UC tables to query and whether to reuse or create a space; don't start by asking for a Genie Space ID. Steps: Genie Guide § Agent workflow.
Read the Genie Guide for configuration, SSE endpoints, and frontend integration.
Common Scaffolding Mistakes
# ❌ WRONG: name is NOT a positional argument
databricks apps init --features analytics my-app-name
# → "unknown command" error
# ✅ CORRECT: use --name flag
databricks apps init --name my-app-name --features analytics --set "..." --profile <PROFILE>
Directory Naming
databricks apps init creates directories in kebab-case matching the app name.
App names must be lowercase with hyphens only (≤26 chars).
Other Frameworks (Streamlit, FastAPI, Flask, Gradio, Dash, Next.js, etc.)
Databricks Apps supports any framework that runs as an HTTP server. LLMs already know these frameworks — the challenge is Databricks platform integration.
READ Other Frameworks Guide BEFORE building any non-AppKit app. It covers port/host configuration, app.yaml and databricks.yml setup, dependency management, networking, and framework-specific gotchas.
Post-Deploy Verification
After deploying, verify the app is running:
databricks apps get <app-name> --profile <PROFILE> -o json # Check app_status.state: RUNNING; the `url` field is the app's URL
databricks apps logs <app-name> --follow --profile <PROFILE> # Stream live logs (Ctrl+C to stop)
Note:
databricks apps logsrequires OAuth authentication and does not work with PAT. Usedatabricks apps getfor status checks if using PAT auth.
Databricks
Core skill for Databricks CLI, authentication, and data exploration.
Product Skills
For specific products, use dedicated skills:
- databricks-jobs - Lakeflow Jobs development and deployment
- databricks-pipelines - Lakeflow Spark Declarative Pipelines (batch and streaming data pipelines)
- databricks-serverless - Compute choice for jobs and pipelines (serverless, performance mode, classic fallback)
- databricks-apps - Full-stack TypeScript app development and deployment
- databricks-lakebase - Lakebase Postgres Autoscaling project management
- databricks-model-serving - Model Serving endpoint management and inference
For data discovery, exploration, and query generation — finding tables, answering natural-language questions about the data, or generating SQL — use databricks-data-discovery (it asks Genie One first, then falls back to manual exploration). If it isn't installed, use the AI-tool commands below and Manual Data Exploration.
Prerequisites
-
CLI installed and current: the CLI must be >= v1.0.0. A CLI that is present but too old is not "good enough" — it must be upgraded, not worked around.
- Run the floor check in CLI Installation. If it reports
UPGRADEorINSTALL: STOP and follow that reference file to upgrade or install — do not proceed or tell the user their CLI is fine. - If another skill routed you here to upgrade (it needs a newer CLI than v1.0.0 — e.g.
databricks-setup-localneeds v1.16.0), that skill already detected the gap. A passing v1.0.0 floor check is not enough: follow the Update / repair procedures to install the latest stable, which satisfies any skill's floor. - Note: In sandboxed environments (Cursor IDE, containers), install commands write outside the workspace and may be blocked. Present the install command to the user and ask them to run it in their own terminal.
- Exception: If CLI installation is blocked (sandboxed containers, restricted environments), ask the user whether to fall back to direct REST API calls using
DATABRICKS_HOSTandDATABRICKS_TOKENenvironment variables if present in the shell. See the Databricks REST API docs.
- Run the floor check in CLI Installation. If it reports
-
Authenticated:
databricks auth profiles- If not: see CLI Authentication
- No account yet? Don't search the web for signup options. Give the user https://www.databricks.com/signup/express?provider=DB&dbx_source=agent-skills&dbxSource=agent-skills&utm_source=agent-skills&utm_campaign=core exactly as written, then continue with
databricks auth login --host <new workspace URL> --profile <name>once they paste the URL.
Profile Selection - CRITICAL
NEVER auto-select a profile.
- List profiles:
databricks auth profiles - Present ALL profiles to user with workspace URLs
- Let user choose (even if only one exists)
- Offer to create new profile if needed
Claude Code - IMPORTANT
Each Bash command runs in a separate shell session.
# WORKS: --profile flag
databricks apps list --profile my-workspace
# WORKS: chained with &&
export DATABRICKS_CONFIG_PROFILE=my-workspace && databricks apps list
# DOES NOT WORK: separate commands
export DATABRICKS_CONFIG_PROFILE=my-workspace
databricks apps list # profile not set!
Tag CLI Commands With Loaded Skills
When databricks --version is v1.27.0 or later, add --skills-invoked <skills> to every databricks CLI command
you run, where <skills> is a comma-separated list of the bare names of the Databricks skills loaded for this task,
for example databricks jobs list --skills-invoked databricks-core,databricks-jobs. Put it right after the
subcommand, before any -- separator, because arguments after -- go to another program (for example
databricks psql --skills-invoked databricks-core,databricks-lakebase --project <ID> -- -c "..."). Use bare skill
names only, with no plugin: prefix. Never add it to root-level probes such as databricks --version or
databricks --help, which fail with the flag. Leave it off on older CLIs, which reject the flag. Do not add it to
scripts or code written for the user.
Data Exploration — Use AI Tools
Use these instead of manually navigating catalogs/schemas/tables:
# discover table structure (columns, types, sample data, stats)
databricks experimental aitools tools discover-schema catalog.schema.table --profile <PROFILE>
# run ad-hoc SQL queries
databricks experimental aitools tools query "SELECT * FROM table LIMIT 10" --profile <PROFILE>
# find the default warehouse
databricks experimental aitools tools get-default-warehouse --profile <PROFILE>
Names are literal. Use catalog/schema/table names exactly as given — never change a
hyphen to an underscore or otherwise normalize them. In SQL, backtick-quote any name part
with special characters (e.g. `my-catalog`.schema.table); unquoted hyphens cause a
parse error.
These commands are first-class for running known SQL and profiling — Genie isn't
required for that. For natural-language data questions, locating data you can't
pin down, or generating a query from a question, prefer the databricks-data-discovery
skill (above) if it's installed. See Manual Data Exploration for the
full command surface, quoting rules, and troubleshooting.
Quick Reference
⚠️ CRITICAL: Some commands use positional arguments, not flags
# current user
databricks current-user me --profile <PROFILE>
# list resources
databricks apps list --profile <PROFILE>
databricks jobs list --profile <PROFILE>
databricks clusters list --profile <PROFILE>
databricks warehouses list --profile <PROFILE>
databricks pipelines list --profile <PROFILE>
databricks serving-endpoints list --profile <PROFILE>
# ⚠️ Unity Catalog — POSITIONAL arguments (NOT flags!)
databricks catalogs list --profile <PROFILE>
# ✅ CORRECT: positional args
databricks schemas list <CATALOG> --profile <PROFILE>
databricks tables list <CATALOG> <SCHEMA> --profile <PROFILE>
databricks tables get <CATALOG>.<SCHEMA>.<TABLE> --profile <PROFILE>
# ❌ WRONG: these flags/commands DON'T EXIST
# databricks schemas list --catalog-name <CATALOG> ← WILL FAIL
# databricks tables list --catalog <CATALOG> ← WILL FAIL
# databricks sql-warehouses list ← doesn't exist, use `warehouses list`
# databricks execute-statement ← doesn't exist, use `experimental aitools tools query`
# databricks sql execute ← doesn't exist, use `experimental aitools tools query`
# When in doubt, check help:
# databricks schemas list --help
# get details
databricks apps get <NAME> --profile <PROFILE>
databricks jobs get --job-id <ID> --profile <PROFILE>
databricks clusters get --cluster-id <ID> --profile <PROFILE>
# bundles
databricks bundle init --profile <PROFILE>
databricks bundle validate --profile <PROFILE>
databricks bundle deploy -t <TARGET> --profile <PROFILE>
databricks bundle run <RESOURCE> -t <TARGET> --profile <PROFILE>
Troubleshooting
| Error | Solution |
|---|---|
cannot configure default credentials | Use --profile flag or authenticate first |
configuration does not support OAuth tokens | The command requires OAuth (e.g., databricks apps logs). Re-authenticate with databricks auth login --host <URL> --profile <PROFILE>. See CLI Authentication. |
PERMISSION_DENIED | Check workspace/UC permissions |
RESOURCE_DOES_NOT_EXIST | Verify resource name/id and profile |
Required Reading by Task
| Task | READ BEFORE proceeding |
|---|---|
| First time setup | CLI Installation |
| Auth issues / new workspace | CLI Authentication |
| Exploring tables/schemas | Manual Data Exploration (or databricks-data-discovery if installed) |
| Deploying jobs/pipelines | Use /databricks-dabs |
Reference Guides
Declarative Automation Bundles (DABs)
Use this skill for any bundle-related request including creating, configuring, validating, deploying, running, and managing Databricks resources through DABs.
Reference Documentation
The following reference files provide detailed guidance for specific bundle tasks:
- Bundle Structure - Bundle structure, databricks.yml configuration, resource definitions, path resolution, variables, and multi-environment targets
- SDP Pipelines - Spark Declarative Pipeline configurations for DABs
- SQL Alerts - SQL Alert schemas and configuration (critical - API differs from other resources)
- Deploy and Run - Validation, deployment, running resources, monitoring logs, and troubleshooting common issues
- Resource Permissions - Permission levels and access control for bundle resources, per-resource-type levels, grants vs permissions
When to Use This Skill
Load this skill for any request involving:
- Creating new bundle projects or resources
- Configuring databricks.yml or resource YAML files
- Setting up multi-environment deployments (dev/prod targets)
- Deploying or running bundle resources
- Managing permissions for bundle resources
- Troubleshooting bundle validation or deployment errors
- Working with specific resource types (dashboards, jobs, pipelines, alerts, volumes, apps)
General Guidelines
- Always validate after configuration changes - Use
bundle validate --strict --target <target>after any change - Use reference documentation - Consult the appropriate reference file for detailed patterns and examples
- Follow naming conventions - Resource files should use
<name>.<resource_type>.ymlformat - Path resolution is critical - Paths differ based on file location (see Bundle Structure reference)
- Preserve existing structure - Keep user comments and structure when editing YAML files
- Use variables - Parameterize catalog, schema, and warehouse for multi-environment support
- Namespace App names - App names are workspace-global and limited to 30 characters. With Databricks CLI 0.270.0 or later, shared-development defaults should include the app and
${workspace.current_user.domain_friendly_name}; override production with a stable name, and persist an explicit local value when the default is invalid, collides, or must distinguish multiple non-production targets in one workspace. On older CLI versions, require an explicitapp_namevalue instead
Required App Completion Contract
Before validating any bundle that creates or changes an App, re-read the final bundle YAML and confirm all of the following structural requirements:
- Each App resource uses a dedicated name variable, such as
name: ${var.app_name}, while preserving an existing equivalent variable when present. - For Databricks CLI 0.270.0 or later, each App name variable defaults to a recognizable App prefix plus
${workspace.current_user.domain_friendly_name}for development, and the production target overrides it with a stable name. On older CLI versions, App name variables have no default and each developer must provide values. - No development App name uses
${workspace.current_user.short_name}, a bare${bundle.target}, or a hardcoded value.
Only after this check, run databricks bundle validate --strict --target <target> --output json and inspect each resolved resources.apps.<key>.name. Each name must contain only lowercase letters, digits, and hyphens and be at most 30 characters. If a resolved name is invalid, collides after normalization, or multiple non-production targets share a workspace, persist a shorter, distinct value in the uncommitted .databricks/bundle/<target>/variable-overrides.json file and validate again. Successful validation alone does not satisfy this contract because validation does not catch every invalid or non-namespaced App name.
Documentation
- Declarative Automation Bundles
- Bundle Examples Repository - official end-to-end example bundles (jobs, pipelines, dashboards, apps, and more); use as working references for patterns not covered by the reference files above
Databricks Data Discovery
This skill routes data work — decide first:
- the request is about the data — finding it, asking questions of it, or
generating SQL → delegate to Genie One:
databricks genie ask -s <session-label> "..."(see Routing below). - writing files or anything else → use your own coding-agent tools.
Genie One just needs an authenticated CLI profile (the parent databricks-core
skill covers auth/profiles if you need it) — but route the data request to Genie
first; don't detour into manual catalog browsing.
Routing — Genie vs your coding agent
Route to Genie when the request is about the data:
- Finding or locating data — "what tables are in wanderbricks?", "where does X live?", which catalog/schema holds something. Let Genie resolve a fuzzy or partial name across all your data instead of guessing or grepping catalogs yourself.
- Questions about the data — "how many…", "what's the average…", "which … the most…", trends, breakdowns.
- Generating SQL from a question — a first-cut query you can run, adapt, or drop into a file.
- Understanding or profiling data to inform a query, dashboard, or app.
Route to your own coding agent (do NOT use Genie) for everything else:
- Writing or editing files — a
.sqlfile, notebook, dashboard, app, config. Genie finds the data and produces the SQL; you write the file. - Authoring jobs, pipelines, apps, or DABs — use those product skills.
- DDL or writes —
CREATE/INSERT/UPDATE/DELETE. - General programming, git, infra, and any non-data task.
Key principle: data discovery, data questions, and query generation → Genie One. Everything else → your coding agent.
Why prefer Genie for data questions
Genie runs inside the Databricks data plane with governed, first-hand access to the org's Unity Catalog metadata, metric views, and curated semantic context — context you do not have when reverse-engineering schemas with ad-hoc SQL. For data questions it is often higher-quality and more performant than doing the discovery yourself, and it keeps improving as a managed Databricks capability. Don't default to writing your own discovery SQL just because you can.
How to ask Genie
Always pass a session label with -s, and prefer reusing the same one: a
follow-up can only continue a conversation if the first ask set the session label,
and reusing it lets later questions build on everything asked so far ("summarize all
of the above"). Use a fresh session label only to start a deliberately separate
session, or distinct session labels to run several in parallel.
The command is databricks genie ask (CLI >= v1.9.0). On an older CLI it lives
under databricks experimental genie ask — same flags and behavior; use that
exact fallback if databricks genie ask is not found.
# Always pass a session label, and reuse the SAME one so follow-ups build on each other
databricks genie ask -s trips "How many bookings were there last week?"
databricks genie ask -s trips "Break that down by destination"
databricks genie ask -s trips "Summarize all of the above"
# --include-sql also prints the SQL Genie ran (use it to generate a query, too)
databricks genie ask -s trips "Write SQL for the top 5 destinations by revenue" --include-sql
# --output json gives a parseable result
databricks genie ask -s trips "Top 5 destinations by revenue" --output json
# → {"status":"completed","conversation_id":"…","text":"…","tool_calls":[{"name":"execute_sql","sql":"…","title":"…"}]}
# Older CLI (< v1.9.0) — same command under the deprecated experimental alias:
# databricks experimental genie ask -s trips "How many bookings were there last week?"
Genie searches across all the data you can see, runs SQL, and streams a grounded
answer — rendered with the executed SQL and, where it helps, a terminal chart. It
auto-resolves a SQL warehouse (override with --warehouse-id); nothing to pick or
set up.
- Streams live: the answer, the agent's steps, and any SQL/results appear as
they arrive. Answers usually take ~5–30s; a stalled stream (no data for ~10 min)
fails with a clear message, and Ctrl-C or
kill(SIGTERM) cancels cleanly. - Picking a session label: any string works — a topic like
trips, or$$for a per-shell session. Default to reusing one session label so follow-ups keep full context; use a fresh one only for a deliberately separate session. An expired session label just starts fresh on the next ask. No id to copy around. - Parallelism: to run sessions at the same time, give each its own session label
(
-s q1,-s q2, …) — independent session labels don't interfere. Within a single session label keep calls sequential: send a follow-up after the previous turn returns, and never fire two asks at once on the same session label (they'd split into two conversations and only one mapping would survive). - Structured output:
--output jsongives{status, conversation_id, text, tool_calls[]}, wheretool_callsincludes the SQL Genie executed;--rawdumps the raw event stream. Note--output jsonbuffers and prints once at the end (no live streaming) — use it for parsing, the default text output for interactive use. - Generating a query: ask Genie to "write SQL for …" and read the SQL from the
response (it's in the answer text, and
--include-sqlalso shows the query Genie ran to verify it — so the SQL is known-good). Genie resolves the schema and joins for you, so this beats hand-writing SQL against unfamiliar tables. - For exact/full rows: Genie shows a preview inline; to pull the complete result
set locally, copy its SQL (
--include-sqlor the JSONtool_calls) into the parent's... aitools tools query "<SQL>". - A non-answer is a message, not an error: if Genie refuses or "couldn't find relevant data," don't retry — use the manual fallback below.
Naming: "Genie One" is the current name for this cross-data chat — formerly "Databricks One", then "OneChat" (the backend tool is still literally named
onechat). All the same thing.
If Genie One isn't available — manual fallback
Only fall back if Genie One is genuinely unavailable — first verify with
databricks genie ask --help (or databricks experimental genie ask --help on a
CLI older than v1.9.0); don't assume the command is missing. When Genie One isn't
enabled, the CLI is too old to have either form of genie ask,
or Genie can't cover the question, do the discovery yourself with the parent skill's
commands — see Manual Data Exploration
(keyword search via information_schema, discover-schema, and tools query).
Running known SQL or profiling a known table that way is perfectly fine on its own.
Do not default to databricks tables list or raw UC REST for data-location
questions — invoke this skill and ask Genie first.
Relationship to the Genie One MCP
Databricks also offers this capability as a managed MCP server — the Genie One MCP. This skill delivers the same functionality through the Databricks CLI, with no MCP server to configure or host. More broadly, the Databricks Agent Skills cover the same ground as Databricks' managed MCP servers, so you don't need any MCP wired up to use this. If you already run the Genie One MCP, use whichever you prefer — they hit the same Genie backend.
Related Skills
- databricks-genie-agents — build and manage Genie Agents: curated agents that let you or a group ask questions of specific data (create, configure, import/export).
- databricks-core (parent) — CLI auth, profiles, and the manual data exploration reference used as the fallback.
Databricks SQL (DBSQL) - Advanced Features
Quick Reference
| Feature | Key Syntax | Since | Reference |
|---|---|---|---|
| SQL Scripting | BEGIN...END, DECLARE, IF/WHILE/FOR | DBR 16.3+ | references/sql-scripting.md |
| Stored Procedures | CREATE PROCEDURE, CALL | DBR 17.0+ | references/sql-scripting.md |
| Recursive CTEs | WITH RECURSIVE | DBR 17.0+ | references/sql-scripting.md |
| Transactions | BEGIN ATOMIC...END | GA | references/sql-scripting.md |
| Materialized Views | CREATE MATERIALIZED VIEW | Pro/Serverless | references/materialized-views-pipes.md |
| Temp Tables | CREATE TEMPORARY TABLE | All | references/materialized-views-pipes.md |
| Pipe Syntax | |> operator | DBR 16.1+ | references/materialized-views-pipes.md |
| Geospatial (H3) | h3_longlatash3(), h3_polyfillash3() | DBR 11.2+ | references/geospatial-collations.md |
| Geospatial (ST) | ST_Point(), ST_Contains(), 80+ funcs | DBR 16.0+ | references/geospatial-collations.md |
| Collations | COLLATE, UTF8_LCASE, locale-aware | DBR 16.1+ | references/geospatial-collations.md |
| AI Functions | ai_query(), ai_classify(), 11+ funcs | DBR 15.1+ | references/ai-functions.md |
| http_request | http_request(conn, ...) | Pro/Serverless | references/ai-functions.md |
| remote_query | SELECT * FROM remote_query(...) | Pro/Serverless | references/ai-functions.md |
| read_files | SELECT * FROM read_files(...) | All | references/ai-functions.md |
| Data Modeling | Star schema, Liquid Clustering | All | references/best-practices.md |
Common Patterns
SQL Scripting - Procedural ETL
BEGIN
DECLARE v_count INT;
DECLARE v_status STRING DEFAULT 'pending';
SET v_count = (SELECT COUNT(*) FROM catalog.schema.raw_orders WHERE status = 'new');
IF v_count > 0 THEN
INSERT INTO catalog.schema.processed_orders
SELECT *, current_timestamp() AS processed_at
FROM catalog.schema.raw_orders
WHERE status = 'new';
SET v_status = 'completed';
ELSE
SET v_status = 'skipped';
END IF;
SELECT v_status AS result, v_count AS rows_processed;
END
Stored Procedure with Error Handling
CREATE OR REPLACE PROCEDURE catalog.schema.upsert_customers(
IN p_source STRING,
OUT p_rows_affected INT
)
LANGUAGE SQL
SQL SECURITY INVOKER
BEGIN
DECLARE EXIT HANDLER FOR SQLEXCEPTION
BEGIN
SET p_rows_affected = -1;
SIGNAL SQLSTATE '45000'
SET MESSAGE_TEXT = concat('Upsert failed for source: ', p_source);
END;
MERGE INTO catalog.schema.dim_customer AS t
USING (SELECT * FROM identifier(p_source)) AS s
ON t.customer_id = s.customer_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;
SET p_rows_affected = (SELECT COUNT(*) FROM identifier(p_source));
END;
-- Invoke:
CALL catalog.schema.upsert_customers('catalog.schema.staging_customers', ?);
Materialized View with Scheduled Refresh
CREATE OR REPLACE MATERIALIZED VIEW catalog.schema.daily_revenue
CLUSTER BY (order_date)
SCHEDULE EVERY 1 HOUR
COMMENT 'Hourly-refreshed daily revenue by region'
AS SELECT
order_date,
region,
SUM(amount) AS total_revenue,
COUNT(DISTINCT customer_id) AS unique_customers
FROM catalog.schema.fact_orders
JOIN catalog.schema.dim_store USING (store_id)
GROUP BY order_date, region;
Pipe Syntax - Readable Transformations
-- Traditional SQL rewritten with pipe syntax
FROM catalog.schema.fact_orders
|> WHERE order_date >= current_date() - INTERVAL 30 DAYS
|> AGGREGATE SUM(amount) AS total, COUNT(*) AS cnt GROUP BY region, product_category
|> WHERE total > 10000
|> ORDER BY total DESC
|> LIMIT 20;
AI Functions - Enrich Data with LLMs
-- Classify support tickets
SELECT
ticket_id,
description,
ai_classify(description, ARRAY('billing', 'technical', 'account', 'feature_request')) AS category,
ai_analyze_sentiment(description) AS sentiment
FROM catalog.schema.support_tickets
LIMIT 100;
-- Extract entities from text
SELECT
doc_id,
ai_extract(content, ARRAY('person_name', 'company', 'dollar_amount')) AS entities
FROM catalog.schema.contracts;
-- General-purpose AI query with structured output
SELECT ai_query(
'databricks-meta-llama-3-3-70b-instruct',
concat('Summarize this customer feedback in JSON with keys: topic, sentiment, action_items. Feedback: ', feedback),
returnType => 'STRUCT<topic STRING, sentiment STRING, action_items ARRAY<STRING>>'
) AS analysis
FROM catalog.schema.customer_feedback
LIMIT 50;
Geospatial - Proximity Search with H3
-- Find stores within 5km of each customer using H3 indexing
WITH customer_h3 AS (
SELECT *, h3_longlatash3(longitude, latitude, 7) AS h3_cell
FROM catalog.schema.customers
),
store_h3 AS (
SELECT *, h3_longlatash3(longitude, latitude, 7) AS h3_cell
FROM catalog.schema.stores
)
SELECT
c.customer_id,
s.store_id,
ST_Distance(
ST_Point(c.longitude, c.latitude),
ST_Point(s.longitude, s.latitude)
) AS distance_m
FROM customer_h3 c
JOIN store_h3 s ON h3_ischildof(c.h3_cell, h3_toparent(s.h3_cell, 5))
WHERE ST_Distance(
ST_Point(c.longitude, c.latitude),
ST_Point(s.longitude, s.latitude)
) < 5000;
Collation - Case-Insensitive Search
-- Create table with case-insensitive collation
CREATE TABLE catalog.schema.products (
product_id BIGINT GENERATED ALWAYS AS IDENTITY,
name STRING COLLATE UTF8_LCASE,
category STRING COLLATE UTF8_LCASE,
price DECIMAL(10, 2)
);
-- Queries automatically case-insensitive (no LOWER() needed)
SELECT * FROM catalog.schema.products
WHERE name = 'MacBook Pro'; -- matches 'macbook pro', 'MACBOOK PRO', etc.
http_request - Call External APIs
-- Set up connection first (one-time)
CREATE CONNECTION my_api_conn
TYPE HTTP
OPTIONS (host 'https://api.example.com', bearer_token secret('scope', 'token'));
-- Call API from SQL
SELECT
order_id,
http_request(
conn => 'my_api_conn',
method => 'POST',
path => '/v1/validate',
json => to_json(named_struct('order_id', order_id, 'amount', amount))
).text AS api_response
FROM catalog.schema.orders
WHERE needs_validation = true;
read_files - Ingest Raw Files
-- Read JSON files from a Volume with schema hints
SELECT *
FROM read_files(
'/Volumes/catalog/schema/raw/events/',
format => 'json',
schemaHints => 'event_id STRING, timestamp TIMESTAMP, payload MAP<STRING, STRING>',
pathGlobFilter => '*.json',
recursiveFileLookup => true
);
-- Read CSV with options
SELECT *
FROM read_files(
'/Volumes/catalog/schema/raw/sales/',
format => 'csv',
header => true,
delimiter => '|',
dateFormat => 'yyyy-MM-dd',
schema => 'sale_id INT, sale_date DATE, amount DECIMAL(10,2), store STRING'
);
Recursive CTE - Hierarchy Traversal
WITH RECURSIVE org_chart AS (
-- Anchor: top-level managers
SELECT employee_id, name, manager_id, 0 AS depth, ARRAY(name) AS path
FROM catalog.schema.employees
WHERE manager_id IS NULL
UNION ALL
-- Recursive: direct reports
SELECT e.employee_id, e.name, e.manager_id, o.depth + 1, array_append(o.path, e.name)
FROM catalog.schema.employees e
JOIN org_chart o ON e.manager_id = o.employee_id
WHERE o.depth < 10 -- safety limit
)
SELECT * FROM org_chart ORDER BY depth, name;
remote_query - Federated Queries
-- Query PostgreSQL via Lakehouse Federation
SELECT *
FROM remote_query(
'my_postgres_connection',
database => 'my_database',
query => 'SELECT customer_id, email, created_at FROM customers WHERE active = true'
);
Reference Files
Load these for detailed syntax, full parameter lists, and advanced patterns:
| File | Contents | When to Read |
|---|---|---|
| references/sql-scripting.md | SQL Scripting, Stored Procedures, Recursive CTEs, Transactions | User needs procedural SQL, error handling, loops, dynamic SQL |
| references/materialized-views-pipes.md | Materialized Views, Temp Tables/Views, Pipe Syntax | User needs MVs, refresh scheduling, temp objects, pipe operator |
| references/geospatial-collations.md | 39 H3 functions, 80+ ST functions, Collation types and hierarchy | User needs spatial analysis, H3 indexing, case/accent handling |
| references/ai-functions.md | 13 AI functions, http_request, remote_query, read_files (all options) | User needs AI enrichment, API calls, federation, file ingestion |
| references/best-practices.md | Data modeling, performance, Liquid Clustering, anti-patterns | User needs architecture guidance, optimization, or modeling advice |
Key Guidelines
- Always use Serverless SQL warehouses for AI functions, MVs, and http_request
- Use
LIMITduring development with AI functions to control costs - Prefer Liquid Clustering over partitioning for new tables (1-4 keys max)
- Use
CLUSTER BY AUTOwhen unsure about clustering keys - Star schema in Gold layer for BI; OBT acceptable in Silver
- Define PK/FK constraints on dimensional models for query optimization
- Use
COLLATE UTF8_LCASEfor user-facing string columns that need case-insensitive search - Test SQL via CLI (
databricks experimental aitools tools query) or notebooks before deploying. If--warehouseis rejected on your CLI version, setDATABRICKS_WAREHOUSE_IDin the environment instead.
Databricks Documentation Reference
This skill provides access to the complete Databricks documentation index via llms.txt - use it as a reference resource to supplement other skills.
Role of This Skill
This is a reference skill, not an action skill. Use it to:
- Look up documentation when other skills don't cover a topic
- Get authoritative guidance on Databricks concepts and APIs
- Find detailed information to inform CLI commands and SDK usage
- Discover features and capabilities you may not know about
Always prefer using CLI/SDK for actions and load specific skills for workflows (databricks-python-sdk, databricks-pipelines, etc.). Use this skill when you need reference documentation.
How to Use
Fetch the llms.txt documentation index:
URL: https://docs.databricks.com/llms.txt
Use WebFetch to retrieve this index, then:
- Search for relevant sections/links
- Fetch specific documentation pages for detailed guidance
- Apply what you learn using the appropriate CLI commands or SDK
Documentation Structure
The llms.txt file is organized by category:
- Overview & Getting Started - Basic concepts and tutorials
- Data Engineering - Lakeflow, Spark, Delta Lake, pipelines
- SQL & Analytics - Warehouses, queries, dashboards
- AI/ML - MLflow, model serving, GenAI
- Governance - Unity Catalog, permissions, security
- Developer Tools - SDKs, CLI, APIs, Terraform
Example: Complementing Other Skills
Scenario: User wants to create a Delta Live Tables pipeline
- Load
databricks-pipelinesskill for workflow patterns - Use this skill to fetch docs if you need clarification on specific DLT features
- Use
databricks pipelines createCLI command to create the pipeline
Scenario: User asks about an unfamiliar Databricks feature
- Fetch llms.txt to find relevant documentation
- Read the specific docs to understand the feature
- Determine which skill/tools apply, then use them
Related Skills
- databricks-python-sdk - SDK patterns for programmatic Databricks access
- databricks-pipelines - DLT / Lakeflow pipeline workflows
- databricks-unity-catalog - Governance and catalog management
- databricks-model-serving - Serving endpoints and model deployment
- databricks-mlflow-evaluation - MLflow agent evaluation and scorers; use
databricks-mlflow-tracingfor tracing and observability
Databricks Execution & Compute
Run code on Databricks. Four execution modes—choose based on workload. Most examples below use the Databricks CLI (Databricks Sandbox is SDK-driven); see the databricks-core skill for install and authentication.
Execution Mode Decision Matrix
| Aspect | Databricks Connect ⭐ | Serverless Job | Interactive Cluster | Databricks Sandbox |
|---|---|---|---|---|
| Use for | Spark code (ETL, data gen) | Heavy processing (ML) | State across tool calls, Scala/R | Untrusted / agent-generated code, isolated env |
| Startup | Instant | ~25-50s cold start | ~5min if stopped | ~1s (warm start) |
| State | Within Python process | None | Via context_id | Home dir persisted across stop/restart |
| Languages | Python (PySpark) | Python, SQL | Python, Scala, SQL, R | Any (general Linux shell) |
| Dependencies | withDependencies() | CLI with environments spec | Install on cluster | Install in sandbox, or spec.environment.image_uri (UC container artifact) for a custom image |
Decision Flow
Main decision point: if you're using Declarative Automation Bundles (DABs) then follow the instructions of the databricks-dabs skill first. In short, you can use databricks bundle run to run code associated with jobs, pipelines, and other resources. This can be recognized by looking for a databricks.yml file in the project root. If these resources don't exist, or if you're not using DABs, then proceed with the below.
Prefer Databricks Connect for all spark-based workload, then serverless.
Spark-based code? → Databricks Connect (fastest)
└─ Python 3.12 missing? → Install it + databricks-connect
└─ Install fails? → Ask user (don't auto-switch modes)
Heavy/long-running (ML)? → Serverless Job (independent)
Need state across calls? → Interactive Cluster (list and ask which one to use)
Scala/R? → Interactive Cluster (list and ask which one to use)
Untrusted / agent-generated code, or need an isolated Linux env? → Databricks Sandbox
How to Run Code
Read the reference file for your chosen mode before proceeding.
Databricks Connect (run locally, prefer when it's pure spark code) → reference
from databricks.connect import DatabricksSession
...
spark = DatabricksSession.builder.profile("my-local-profile").serverless(True).getOrCreate()
python my_spark_script.py
Serverless Job → reference
Pure CLI flow: upload a local file as a workspace notebook, fire a one-time run with databricks jobs submit (create + run in one call, ephemeral — no Jobs UI entry, no retry), then poll + fetch the result. The local file must be a Databricks source notebook — top line # Databricks notebook source (Python) or -- Databricks notebook source (SQL).
1. Upload the local file as a workspace notebook. TARGET_PATH is positional; --file is the local path.
databricks workspace import /Workspace/Users/<user>/.ai_dev_kit/train --file /local/path/to/train.py --format SOURCE --language PYTHON --overwrite
2. Submit the run. Use --no-wait to get {"run_id": N} back immediately; drop it to block until terminated. "client": "4" is required for dependencies to install ("1" silently ignores them).
databricks jobs submit --no-wait --json @submit.json
{
"run_name": "train-run",
"tasks": [{
"task_key": "main",
"notebook_task": {"notebook_path": "/Workspace/Users/<user>/.ai_dev_kit/train"},
"environment_key": "ml_env"
}],
"environments": [{
"environment_key": "ml_env",
"spec": {"client": "4", "dependencies": ["scikit-learn==1.5.2", "mlflow==2.22.0"]}
}]
}
3. Check state / wait for completion. Life-cycle: PENDING → RUNNING → TERMINATED (or SKIPPED / INTERNAL_ERROR). Only read .state.result_state (SUCCESS / FAILED / CANCELED) once life-cycle is TERMINATED.
databricks jobs get-run <RUN_ID> | jq '{state: .state.life_cycle_state, result: .state.result_state, duration_ms: .execution_duration, url: .run_page_url, task_run_id: .tasks[0].run_id}'
4. Fetch the output / error. Gotcha: get-run-output takes the task run_id (.tasks[0].run_id), NOT the parent run_id from submit. notebook_output.result is the string passed to dbutils.notebook.exit().
databricks jobs get-run-output <TASK_RUN_ID> | jq '{result: .notebook_output.result, error, error_trace}'
Always use dbutils.notebook.exit(<string>) in the notebook — print() is not captured by get-run-output. For JSON results: dbutils.notebook.exit(json.dumps({...})) then parse .notebook_output.result client-side.
Interactive Cluster → reference
Avoid by default — prefer Serverless Job. Only use an interactive cluster when:
- you have an existing classic cluster already running and available, or
- you need live, stateful execution across multiple calls (debugging via an execution context), or
- the user explicitly asks for it.
Interactive clusters are slow to start (3-8 min) and cost money while running. Don't start one implicitly.
Databricks Sandbox → skill
Isolated serverless environment for running untrusted or agent-generated code, driven through the Databricks SDK. Prefer it for agent tool-execution and ephemeral build/test environments rather than Spark workloads. See the databricks-sandbox skill for lifecycle, command execution, and file operations.
CLI Command Map
All compute lifecycle and code-execution actions go through the Databricks CLI. Headline commands:
| Action | Command |
|---|---|
| Upload local file as workspace notebook | databricks workspace import <WORKSPACE_PATH> --file <LOCAL> --format SOURCE --language PYTHON --overwrite |
| Run serverless code (upload + submit + wait) | databricks jobs submit --json @submit.json (see Serverless Job section above; with --no-wait for async) |
| Get run state / wait | databricks jobs get-run <RUN_ID> (poll .state.life_cycle_state) |
| Fetch run output | databricks jobs get-run-output <TASK_RUN_ID> |
| List clusters | databricks clusters list --output json |
| Get cluster details | databricks clusters get <CLUSTER_ID> |
| Start / restart / terminate cluster | databricks clusters start/restart/delete <CLUSTER_ID> |
| Permanently delete cluster | databricks clusters permanent-delete <CLUSTER_ID> |
| Create cluster | databricks clusters create --json '{...}' (see 3-interactive-cluster.md) |
| List node types / Spark versions | databricks clusters list-node-types / databricks clusters spark-versions |
| Execute code on a running cluster | databricks api post /api/1.2/contexts/create + databricks api post /api/1.2/commands/execute (see 3-interactive-cluster.md) |
| SQL warehouses | databricks warehouses create/list/get/start/stop/edit/delete (see SQL Warehouses below) |
SQL Warehouses
All ID-taking commands use positional arg (no --id flag). Use databricks warehouses list to find an ID.
# Create a serverless SQL warehouse. min_num_clusters + max_num_clusters are REQUIRED
# (the server rejects the default 0). Keep the aidevkit_project tag for resource tracking.
databricks warehouses create --json '{
"name": "my-warehouse",
"cluster_size": "Small",
"enable_serverless_compute": true,
"auto_stop_mins": 10,
"min_num_clusters": 1,
"max_num_clusters": 1,
"tags": {"custom_tags": [{"key": "aidevkit_project", "value": "ai-dev-kit"}]}
}'
# List / find — trim to id, name, state with jq
databricks warehouses list -o json | jq '.[] | {id, name, state, size: .cluster_size}'
# Find by name
databricks warehouses list -o json | jq '.[] | select(.name == "my-warehouse")'
# Get one warehouse's full config
databricks warehouses get <WAREHOUSE_ID>
# Start / stop (both are LROs; add --no-wait to return immediately)
databricks warehouses start <WAREHOUSE_ID>
databricks warehouses stop <WAREHOUSE_ID>
# Resize / reconfigure — pass the FULL desired config (omitted fields revert to defaults,
# so always re-state min_num_clusters/max_num_clusters). Use --no-wait if the warehouse
# is STOPPED, otherwise edit blocks trying to reach RUNNING and errors out (the mutation
# itself still applies). When the warehouse is already RUNNING, --no-wait is optional.
databricks warehouses edit <WAREHOUSE_ID> --no-wait --json '{
"name": "my-warehouse",
"cluster_size": "Medium",
"enable_serverless_compute": true,
"auto_stop_mins": 15,
"min_num_clusters": 1,
"max_num_clusters": 1
}'
# Delete (irreversible)
databricks warehouses delete <WAREHOUSE_ID>
Sizes: 2X-Small, X-Small, Small, Medium, Large, X-Large, 2X-Large, 3X-Large, 4X-Large. Types: set "warehouse_type": "PRO" (default) or "CLASSIC" in the JSON body.
Related Skills
- databricks-synthetic-data-gen — Data generation using Spark + Faker
- databricks-jobs — Production job orchestration
- databricks-dbsql — SQL warehouse and AI functions
- databricks-sandbox — Isolated serverless sandboxes for untrusted / agent-generated code
Databricks Genie Agents
A Genie Agent is a curated agent scoped to one business area: its tables / Metric Views, documents (UC volumes), sample questions, and instructions. It answers by writing SQL on a warehouse and, in Agent mode, by searching its documents, so one turn can combine SQL results and cited files. Distinct from Genie One (databricks-data-discovery), which answers across your data without a curated Agent.
Replaces Knowledge Assistant and Supervisor Agent for docs + data. Both are legacy ("For new low-code agents over your documents and data, use Genie Agents"). Document Q&A (was a KA) = an Agent with a volume: no endpoint, no sync. Routing between data and docs (was a Supervisor over Genie + KA) = one Agent with tables + a volume. See Documents. An Agent can also attach Databricks-provided MCP connectors (Google Drive, Microsoft 365, Slack, Jira/Confluence, GitHub, ...; Beta, set up in the UI, Agent mode only). Custom MCP servers, UC functions, or other endpoints aren't supported; build a code agent (
databricks-ml-training). To maintain an existing KA / Supervisor: legacy-ka-supervisor.md.
Genie Agent Lifecycle
| Phase | Reference | Load when | Typical CLI |
|---|---|---|---|
| Create / update | create-genie-agent.md | Building or editing an Agent: the workflow, the serialized_space payload format, and the CLI | discover-schema → create-space / update-space |
| Attach documents | create-genie-agent.md § Documents | Document Q&A, docs + data, replacing a KA / Supervisor | volumes create → update-space → probe in Agent mode |
| Query / converse / validate | query-genie-agent.md | Conversation API (tables), Agent mode API (needed for documents), Metric View SQL | start-conversation / Agent mode responses |
| Improve (inspect / diagnose / benchmark / fix) | improve-genie-agent.md | Working with an existing Agent: wrong/empty answers, quality tuning, benchmark evaluation | get-space; genie-create-eval-run; update-space |
| Export / migrate | genie-agent-cicd.md | Export, import, cross-workspace migration, batch migration, DABs/CI-CD | get-space → remap → create-space |
| Optimization history | uc-persistence.md | Setting up UC Delta tables for multi-pass optimization history (CREATE TABLE DDL only) | |
| Legacy | legacy-ka-supervisor.md | Maintaining or migrating an existing Knowledge Assistant / Supervisor Agent | knowledge-assistants / supervisor-agents |
Typical flow: create → query/validate → improve (diagnose / benchmark / fix, as needed).
Commands
An Agent is defined by a serialized_space JSON object (sources, sample questions, per-column context, instructions), kept in a local genie_agent.json and passed to the CLI. Core commands below; the lifecycle table points to the full workflow, payload schema, and flags.
databricks genie list-spaces # find Agents
databricks genie get-space SPACE_ID --include-serialized-space # inspect config
databricks genie create-space WAREHOUSE_ID "$(jq -c . genie_agent.json)" \
--title "..." --parent-path /Workspace/Users/you@company.com/genie_spaces # create
databricks genie update-space SPACE_ID --serialized-space "$(jq -c . genie_agent.json)" # full-replace edit
databricks genie start-conversation --no-wait SPACE_ID "a question" # converse / validate (tables only)
databricks genie trash-space SPACE_ID # delete
Documents (replaces KA and Supervisor)
Attach a managed UC volume in data_sources.volumes[]: create-space / update-space start indexing automatically, and documents are read in Agent mode only (the chat-mode start-conversation ignores them). Setup, waiting for indexing, adding documents to an existing Agent, permissions, limits, and evaluation: create-genie-agent.md § Documents.
Prerequisites
- Tables / Metric Views in Unity Catalog and/or a managed UC volume of documents
- SQL Warehouse to execute queries (auto-detected if not specified)
Related Skills
- databricks-data-discovery: general natural-language data exploration / "ask Genie" (Genie One) across your data, when you are not targeting a specific curated Genie Agent
- databricks-metric-views: governed business metrics that Genie consumes. See its SKILL.md for design rules that affect Genie answer quality, and query-patterns.md for the
MEASURE()query rules Genie follows. - databricks-unstructured-pdf-generation: generate demo / eval PDFs to attach to an Agent as a volume
- databricks-ml-training: code agents (
ResponsesAgent) when you need orchestration Genie can't express - databricks-unity-catalog: manage the catalogs, schemas, tables, and volumes Genie uses
Apache Iceberg on Databricks
Databricks provides multiple ways to work with Apache Iceberg: native managed Iceberg tables, UniForm for Delta-to-Iceberg interoperability, and the Iceberg REST Catalog (IRC) for external engine access.
Critical Rules (always follow)
- MUST use Unity Catalog — all Iceberg features require UC-enabled workspaces
- MUST NOT install an Iceberg library into Databricks Runtime (DBR includes built-in Iceberg support; adding a library causes version conflicts)
- MUST NOT set
write.metadata.pathorwrite.metadata.previous-versions-max— Databricks manages metadata locations automatically; overriding causes corruption - MUST determine which Iceberg pattern fits the use case before writing code — see the When to Use section below
- MUST know that both
PARTITIONED BYandCLUSTER BYproduce the same Iceberg metadata for external engines — UC maintains an Iceberg partition spec with partition fields corresponding to the clustering keys, so external engines reading via IRC see a partitioned Iceberg table (not Hive-style, but proper Iceberg partition fields) and can prune on those fields; internally UC uses those fields as liquid clustering keys; the only differences between the two syntaxes are: (1)PARTITIONED BYis standard Iceberg DDL (any engine can create the table), whileCLUSTER BYis DBR-only DDL; (2)PARTITIONED BYauto-handles DV/row-tracking properties, whileCLUSTER BYrequires manual TBLPROPERTIES on v2 - MUST NOT use expression-based partition transforms (
bucket(),years(),months(),days(),hours()) withPARTITIONED BYon managed Iceberg tables — only plain column references are supported; expression transforms cause errors - MUST disable deletion vectors and row tracking when using
CLUSTER BYon Iceberg v2 tables — set'delta.enableDeletionVectors' = falseand'delta.enableRowTracking' = falsein TBLPROPERTIES (Iceberg v3 handles this automatically;PARTITIONED BYhandles this automatically on both v2 and v3)
Key Concepts
| Concept | Summary |
|---|---|
| Managed Iceberg Table | Native Iceberg table created with USING ICEBERG — full read/write in Databricks and via external Iceberg engines |
| External Iceberg Reads (Uniform) | Delta table that auto-generates Iceberg metadata — read as Iceberg externally, write as Delta internally |
| Compatibility Mode | UniForm variant for streaming tables and materialized views in SDP pipelines |
| Iceberg REST Catalog (IRC) | Unity Catalog's built-in REST endpoint implementing the Iceberg REST Catalog spec — lets external engines (Spark, PyIceberg, Snowflake) access UC-managed Iceberg data |
| Iceberg v3 | Next-gen format (DBR 18 LTS+) — deletion vectors, VARIANT type, row lineage |
Quick Start
Create a Managed Iceberg Table
-- No clustering
CREATE TABLE my_catalog.my_schema.events
USING ICEBERG
AS SELECT * FROM raw_events;
-- PARTITIONED BY (recommended for cross-platform): standard Iceberg syntax, works on EMR/OSS Spark/Trino/Flink
-- auto-disables DVs and row tracking — no TBLPROPERTIES needed on v2 or v3
CREATE TABLE my_catalog.my_schema.events
USING ICEBERG
PARTITIONED BY (event_date)
AS SELECT * FROM raw_events;
-- CLUSTER BY on Iceberg v2 (DBR-only syntax): must manually disable DVs and row tracking
CREATE TABLE my_catalog.my_schema.events
USING ICEBERG
TBLPROPERTIES (
'delta.enableDeletionVectors' = false,
'delta.enableRowTracking' = false
)
CLUSTER BY (event_date)
AS SELECT * FROM raw_events;
-- CLUSTER BY on Iceberg v3 (DBR-only syntax): no TBLPROPERTIES needed
CREATE TABLE my_catalog.my_schema.events
USING ICEBERG
TBLPROPERTIES ('format-version' = '3')
CLUSTER BY (event_date)
AS SELECT * FROM raw_events;
Enable UniForm on an Existing Delta Table
ALTER TABLE my_catalog.my_schema.customers
SET TBLPROPERTIES (
'delta.columnMapping.mode' = 'name',
'delta.enableIcebergCompatV2' = 'true',
'delta.universalFormat.enabledFormats' = 'iceberg'
);
Read/Write Capability Matrix
| Table Type | Databricks Read | Databricks Write | External IRC Read | External IRC Write |
|---|---|---|---|---|
Managed Iceberg (USING ICEBERG) | Yes | Yes | Yes | Yes |
| Delta + UniForm | Yes (as Delta) | Yes (as Delta) | Yes (as Iceberg) | No |
| Delta + Compatibility Mode | Yes (as Delta) | Yes | Yes (as Iceberg) | No |
Reference Files
| File | Summary | Keywords |
|---|---|---|
| references/1-managed-iceberg-tables.md | Creating and managing native Iceberg tables — DDL, DML, Liquid Clustering, Predictive Optimization, Iceberg v3, limitations | CREATE TABLE USING ICEBERG, CTAS, MERGE, time travel, deletion vectors, VARIANT |
| references/2-uniform-and-compatibility.md | Making Delta tables readable as Iceberg — UniForm for regular tables, Compatibility Mode for streaming tables and MVs | UniForm, universalFormat, Compatibility Mode, streaming tables, materialized views, SDP |
| references/3-iceberg-rest-catalog.md | Exposing Databricks tables to external engines via the IRC endpoint — auth, credential vending, IP access lists | IRC, REST Catalog, credential vending, EXTERNAL USE SCHEMA, PAT, OAuth |
| references/4-snowflake-interop.md | Bidirectional Snowflake-Databricks integration — catalog integration, foreign catalogs, vended credentials | Snowflake, catalog integration, external volume, vended credentials, REFRESH_INTERVAL_SECONDS |
| references/5-external-engine-interop.md | Connecting PyIceberg, OSS Spark, AWS EMR, Apache Flink, and Kafka Connect via IRC | PyIceberg, OSS Spark, EMR, Flink, Kafka Connect, pyiceberg.yaml |
When to Use
- Creating a new Iceberg table → references/1-managed-iceberg-tables.md
- Making an existing Delta table readable as Iceberg → references/2-uniform-and-compatibility.md
- Making a streaming table or MV readable as Iceberg → references/2-uniform-and-compatibility.md (Compatibility Mode section)
- Choosing between Managed Iceberg vs UniForm vs Compatibility Mode → decision table in references/2-uniform-and-compatibility.md
- Exposing Databricks tables to external engines via REST API → references/3-iceberg-rest-catalog.md
- Integrating Databricks with Snowflake (either direction) → references/4-snowflake-interop.md
- Connecting PyIceberg, OSS Spark, Flink, EMR, or Kafka → references/5-external-engine-interop.md
Common Issues
| Issue | Solution |
|---|---|
| No Change Data Feed (CDF) | CDF is not supported on managed Iceberg tables. Use Delta + UniForm if you need CDF. |
| UniForm async delay | Iceberg metadata generation is asynchronous. After a write, there may be a brief delay before external engines see the latest data. Check status with DESCRIBE EXTENDED table_name. |
| Compression codec change | Managed Iceberg tables use zstd compression by default (not snappy). Older Iceberg readers that don't support zstd will fail. Verify reader compatibility or set write.parquet.compression-codec to snappy. |
| Snowflake 1000-commit limit | Snowflake's Iceberg catalog integration can only see the last 1000 Iceberg commits. High-frequency writers must compact metadata or Snowflake will lose visibility of older data. |
| Deletion vectors with UniForm | UniForm requires deletion vectors to be disabled (delta.enableDeletionVectors = false). If your table has deletion vectors enabled, disable them before enabling UniForm. |
| No shallow clone for Iceberg | SHALLOW CLONE is not supported for Iceberg tables. Use DEEP CLONE or CREATE TABLE ... AS SELECT instead. |
| Version mismatch with external engines | Ensure external engines use an Iceberg library version compatible with the format version of your tables. Iceberg v3 tables require Iceberg library 1.9.0+. |
Related Skills
- databricks-unity-catalog — catalog/schema management, governance, system tables
- databricks-pipelines — SDP pipelines (streaming tables, materialized views with Compatibility Mode)
- databricks-python-sdk — Python SDK and REST API for Databricks operations
- databricks-dbsql — SQL warehouse features, query patterns
Resources
- Iceberg Overview — main hub for Iceberg on Databricks
- UniForm — Delta Universal Format
- Iceberg REST Catalog — IRC endpoint and external engine access
- Compatibility Mode — UniForm for streaming tables and MVs
- Iceberg v3 — next-gen format features
- Foreign Tables — reading external catalog data
Lakeflow Jobs Development
FIRST: Use the parent databricks-core skill for CLI basics, authentication, profile selection, and data exploration commands.
Compute: jobs run on serverless by default. Before you choose compute or write any cluster setting, use the
databricks-serverless skill; it decides serverless or classic and the performance mode.
Lakeflow Jobs orchestrate data workflows with multi-task DAGs, flexible triggers, and comprehensive monitoring. Jobs support diverse task types and can be managed via Declarative Automation Bundles (DABs), Python SDK, or CLI.
Reference Files
| Use Case | Reference File |
|---|---|
| Configure task types (notebook, Python, SQL, dbt, pipeline, JAR, run_job, for_each) | references/task-types.md |
| Set up triggers and schedules (cron, periodic, file arrival, table update, continuous) | references/triggers-schedules.md |
| Configure notifications, health rules, retries, timeouts, queues | references/notifications-monitoring.md |
| Complete worked examples (ETL, warehouse refresh, event-driven, ML training, multi-env, streaming, cross-job) | references/examples.md |
Scaffolding a New Job Project
Use databricks bundle init with a config file to scaffold non-interactively. This creates a project in the <project_name>/ directory:
databricks bundle init default-python --config-file <(echo '{"project_name": "my_job", "include_job": "yes", "include_pipeline": "no", "include_python": "yes", "serverless": "yes"}') --profile <PROFILE> < /dev/null
project_name: letters, numbers, underscores only
bundle init also creates an AGENTS.md in the project. Customize that file for the project
instead of creating new agent instruction files.
Project Structure
my-job-project/
├── databricks.yml # Bundle configuration
├── resources/
│ └── my_job.job.yml # Job definition
├── src/
│ ├── my_notebook.ipynb # Notebook tasks
│ └── my_module/ # Python wheel package
│ ├── __init__.py
│ └── main.py
├── tests/
│ └── test_main.py
└── pyproject.toml # Python project config (if using wheels)
Quick Start
Declarative Automation Bundles (DABs) — recommended
# resources/jobs.yml
resources:
jobs:
my_etl_job:
name: "[${bundle.target}] My ETL Job"
tasks:
- task_key: extract
notebook_task:
notebook_path: ../src/notebooks/extract.py
# no cluster block = serverless; compute details: databricks-serverless
Python SDK
from databricks.sdk import WorkspaceClient
from databricks.sdk.service.jobs import Task, NotebookTask, Source
w = WorkspaceClient()
job = w.jobs.create(
name="my-etl-job",
tasks=[
Task(
task_key="extract",
notebook_task=NotebookTask(
notebook_path="/Workspace/Shared/etl/extract",
source=Source.WORKSPACE,
),
),
],
)
print(f"Created job: {job.job_id}")
CLI
databricks jobs create --json '{
"name": "my-etl-job",
"tasks": [{
"task_key": "extract",
"notebook_task": {
"notebook_path": "/Workspace/Shared/etl/extract",
"source": "WORKSPACE"
}
}]
}'
Core Concepts
Multi-Task Workflows
Jobs support DAG-based task dependencies:
tasks:
- task_key: extract
notebook_task:
notebook_path: ../src/extract.py
- task_key: transform
depends_on:
- task_key: extract
notebook_task:
notebook_path: ../src/transform.py
- task_key: load
depends_on:
- task_key: transform
run_if: ALL_SUCCESS # Only run if all dependencies succeed
notebook_task:
notebook_path: ../src/load.py
run_if conditions:
ALL_SUCCESS(default) — run when all dependencies succeedALL_DONE— run when all dependencies complete (success or failure)AT_LEAST_ONE_SUCCESS— run when at least one dependency succeedsNONE_FAILED— run when no dependencies failedALL_FAILED— run when all dependencies failedAT_LEAST_ONE_FAILED— run when at least one dependency failed
Task Types Summary
| Task Type | Use Case | Reference |
|---|---|---|
notebook_task | Run notebooks | references/task-types.md#notebook-task |
spark_python_task | Run Python scripts | references/task-types.md#spark-python-task |
python_wheel_task | Run Python wheels | references/task-types.md#python-wheel-task |
sql_task | Run SQL queries/files/dashboards/alerts | references/task-types.md#sql-task |
dbt_task | Run dbt projects | references/task-types.md#dbt-task |
pipeline_task | Trigger SDP (formerly DLT) pipelines | references/task-types.md#pipeline-task |
spark_jar_task | Run Spark JARs | references/task-types.md#spark-jar-task |
run_job_task | Trigger other jobs | references/task-types.md#run-job-task |
for_each_task | Loop over inputs | references/task-types.md#for-each-task |
Trigger Types Summary
| Trigger Type | Use Case | Reference |
|---|---|---|
schedule | Cron-based scheduling | references/triggers-schedules.md#cron-schedule |
trigger.periodic | Interval-based | references/triggers-schedules.md#periodic-trigger |
trigger.file_arrival | File arrival events | references/triggers-schedules.md#file-arrival-trigger |
trigger.table_update | Unity Catalog table change events | references/triggers-schedules.md#table-update-trigger |
continuous | Always-running jobs | references/triggers-schedules.md#continuous-jobs |
Compute Configuration
Serverless is the default. Leave out cluster settings (job_clusters, new_cluster, job_cluster_key,
existing_cluster_id); a notebook, Python script, wheel or JAR task without them runs on serverless.
SQL tasks, and the SQL of dbt tasks, run on a SQL warehouse (warehouse_id, see
Task Types); a pipeline task runs on the pipeline's own compute
(serverless: true in the pipeline), and the job's performance_target sets its mode.
If creating or running the job fails because serverless is not available in the workspace, put the
whole job on classic and say so (job clusters for the tasks that run on job compute).
For everything else about compute, use the databricks-serverless skill: the performance mode
(performance_target), libraries (environments), existing code that may not run on serverless, and
the few workloads that need a classic cluster. If that skill is not installed, run
databricks aitools install. Either way, set performance_target on every job: STANDARD to optimize
cost, PERFORMANCE_OPTIMIZED when the job is time-sensitive, interactive, or other workloads depend on it.
Job Parameters
Parameters defined at job level are passed to ALL tasks (no need to repeat per task):
parameters:
- name: env
default: "dev"
- name: date
default: "{{start_date}}" # Dynamic value reference
Access in notebooks:
catalog = dbutils.widgets.get("env")
load_date = dbutils.widgets.get("date")
Pass to specific tasks:
tasks:
- task_key: my_task
notebook_task:
notebook_path: ../src/notebook.py
base_parameters:
env: "{{job.parameters.env}}"
custom_param: "value"
Common Operations
Python SDK
from databricks.sdk import WorkspaceClient
w = WorkspaceClient()
# List jobs
jobs = w.jobs.list()
# Get job details
job = w.jobs.get(job_id=12345)
# Run job now
run = w.jobs.run_now(job_id=12345)
# Run with parameters
run = w.jobs.run_now(
job_id=12345,
job_parameters={"env": "prod", "date": "2024-01-15"},
)
# Cancel run
w.jobs.cancel_run(run_id=run.run_id)
# Delete job
w.jobs.delete(job_id=12345)
CLI
# List jobs
databricks jobs list
# Get job details
databricks jobs get 12345
# Run job
databricks jobs run-now 12345
# Run with parameters (must use --json with job_id inside)
databricks jobs run-now --json '{"job_id": 12345, "job_parameters": {"env": "prod"}}'
# Cancel run
databricks jobs cancel-run 67890
# Delete job
databricks jobs delete 12345
Declarative Automation Bundle Operations
# Validate configuration
databricks bundle validate --profile <profile>
# Deploy to a target
databricks bundle deploy -t dev --profile <profile>
# Run a job
databricks bundle run <job_name> -t dev --profile <profile>
# Check run status
databricks jobs get-run --run-id <id> --profile <profile>
# Destroy resources
databricks bundle destroy --auto-approve
Permissions (DABs)
resources:
jobs:
my_job:
name: "My Job"
permissions:
- level: CAN_VIEW
group_name: "data-analysts"
- level: CAN_MANAGE_RUN
group_name: "data-engineers"
- level: CAN_MANAGE
user_name: "admin@example.com"
Permission levels:
CAN_VIEW— view job and run historyCAN_MANAGE_RUN— view, trigger, and cancel runsCAN_MANAGE— full control including edit and delete
Unit Testing
Run unit tests locally:
uv run pytest
Development Workflow
- Validate:
databricks bundle validate --profile <profile> - Deploy:
databricks bundle deploy -t dev --profile <profile> - Run:
databricks bundle run <job_name> -t dev --profile <profile> - Check run status:
databricks jobs get-run --run-id <id> --profile <profile>
Common Issues
| Issue | Solution |
|---|---|
| Serverless job startup slow | performance_target: PERFORMANCE_OPTIMIZED; see databricks-serverless |
| Task dependencies not working | Verify task_key references match exactly in depends_on |
| Schedule not triggering | Check pause_status: UNPAUSED and valid timezone |
| File arrival not detecting | Ensure path has proper permissions and uses cloud storage URL |
| Table update trigger missing events | Verify Unity Catalog table and proper grants |
| Parameter not accessible | Use dbutils.widgets.get() in notebooks |
admins group error | Cannot modify admins permissions on jobs |
| Serverless task fails | See the Common Issues of databricks-serverless (libraries, RDD code, DBFS mounts) |
Related Skills
- databricks-dabs — DABs configuration patterns shared by jobs and pipelines
- databricks-pipelines — SDP (formerly DLT) pipelines triggered by
pipeline_task - databricks-serverless — compute decisions: serverless, performance mode, classic fallback
Documentation
Lakebase Postgres Autoscaling
FIRST: Use the parent databricks-core skill for CLI basics, authentication, and profile selection.
Lakebase is Databricks' serverless Postgres-compatible database, available on AWS and Azure (GA) and on GCP (Beta). It provides fully managed OLTP storage with autoscaling, branching, and scale-to-zero.
Regions: Lakebase runs only in the commercial regions listed in Region availability (check the page for the user's cloud). It isn't available on AWS GovCloud, and the Azure list has no Azure Government or Azure China regions. If the workspace is in one of these clouds or an unlisted region, tell the user that Lakebase isn't available there. Don't try to create a project.
Provisioned Lakebase no longer exists. Never create it. The old Provisioned tier (static, non-scalable Postgres;
databricks databaseCLI /w.databaseSDK) is being migrated to Autoscaling over summer 2026, with no customer action required. Lakebase today is Autoscaling-only (databricks postgres/w.postgres). Do not attempt to create Provisioned instances; if a user asks for one, tell them Provisioned instances don't exist anymore and point them to Autoscaling. Treat the legacydatabaseresource key in app configs as retired, and migrate to thepostgresresource key. For the legacy DABsynced_database_tablesresource, use thepostgres_synced_tablesbundle resource or thedatabricks postgres create-synced-tableCLI flow instead (see the Common Issues row andreferences/synced-tables.md).
For context when reading older configs or docs, here is how the retired Provisioned tier maps to Autoscaling:
| Aspect | Provisioned (retired) | Autoscaling (current) |
|---|---|---|
| CLI group | databricks database | databricks postgres |
| Top-level resource | Instance | Project |
| Capacity | CU_1–CU_8 (16 GB/CU) | 0.5–112 CU (2 GB/CU) |
| Branching | Not supported | Full support |
| Scale-to-zero | Not supported | Configurable |
| HA | Readable secondaries | 1–3 secondaries + read replicas |
| Data API | Not available | PostgREST HTTP API |
| Cloud | AWS only | AWS and Azure (GA), GCP (Beta) |
Compliance: Supports HIPAA, C5, TISAX, or None. Before any PHI goes into Lakebase, apply the HIPAA gate below.
HIPAA Gate (PHI)
HIPAA coverage comes from the workspace: the compliance security profile with the HIPAA standard, plus a Business Associate Agreement. Confirm both with the user before PHI goes in. Then:
- Preview features: do not use a Preview or Beta feature (for example, Lakehouse Sync) for PHI unless it is listed for your region in Regional support for features. Check the list each time.
- Existing projects: a project that was running when HIPAA was turned on has no HIPAA audit logging until its compute restarts. Databricks does not restart it for you. Restart each compute.
- No PHI in metadata: never put PHI in database, schema, table, column, or role names, in tags, or in other customer-defined fields. These appear in audit logs and system metadata, and Databricks cannot remove them later.
- No PHI in SQL text: pass PHI values as query parameters, never as literals. Audit logs record statement text.
- No PHI in logs or tickets: keep PHI out of app logs, error messages, screenshots, diagnostic output, and support requests.
- Least privilege: grant each role only the schemas, tables, and operations it needs.
Sources: HIPAA compliance for Lakebase, Enable HIPAA compliance.
Row-Level Security and Data API: Must-Follow Rules
Apply these whenever a design has per-user rows, RLS, or the Data API. Details and tests are in connectivity.md.
- Many end users: a branch holds at most 500 Postgres roles. Never create one role per app user. The app connects as its service principal over a pool and sets the user per transaction:
SELECT set_config('app.user', $1, true). - RLS on a setting: wrap it in
NULLIF:USING (owner_email = (SELECT NULLIF(current_setting('app.user', true), ''))). On a reused pooled connection, an unset value reads'', not NULL. Also runALTER TABLE ... FORCE ROW LEVEL SECURITY, because the table owner bypasses RLS otherwise. - Testing RLS: never test as the project owner or
databricks_superuser, and never grantdatabricks_superuserto a teammate for local work. Both bypass RLS. Use a role with onlyUSAGEon the app's schema and the table privileges it needs. - Views over RLS tables: create them
WITH (security_invoker = true), and grant the callerSELECTon the base table too. A default view checks RLS as its owner, so every caller sees all rows. - Data API before production: the defaults are open. Empty CORS origins means all origins, and empty maximum rows means no limit. Set the exact app origins and a row limit, and expose only the schema built for the API, not
public.
Capabilities
- Project lifecycle -- create, update, delete Lakebase Postgres Autoscaling projects
- Branching -- copy-on-write branches with TTL, point-in-time recovery, and reset
- Compute scaling -- autoscale 0.5--64 CU, fixed up to 112 CU, scale-to-zero
- High availability -- 1 primary + 1--3 secondaries, automatic failover
- PostgreSQL connectivity -- OAuth token refresh, connection pooling, SSL
- Data API -- PostgREST-compatible HTTP CRUD (Autoscaling only)
- Lakebase synced tables -- sync Unity Catalog Delta tables into Postgres (previously known as Reverse ETL)
- Databricks App integration -- scaffold apps with Lakebase feature, deploy-first workflow
- Cloud support -- AWS and Azure (GA), GCP (Beta)
Reference docs:
- computes-and-scaling.md — Sizing, endpoint management, scale-to-zero, HA
- connectivity.md — Connection patterns, token refresh, Data API
- synced-tables.md — Lakebase synced tables, data type mapping, capacity planning
- lakehouse-sync.md — CDC from Lakebase Postgres to Unity Catalog Delta tables (programmable via
databricks postgres *-cdf-config/w.postgres.*_cdf_config, Beta) - pgvector.md — Vector similarity search with pgvector extension
- off-platform.md — Off-platform Lakebase (NOT Databricks Apps): external Node.js apps connecting via
@databricks/lakebase, env management, token refresh, Drizzle ORM
Resource Hierarchy
Project (top-level container)
└── Branch (isolated database environment, copy-on-write)
├── Endpoint (read-write or read-only)
├── Database (standard Postgres DB)
└── Role (Postgres role)
- Project: Top-level container. Creating one auto-provisions a
productionbranch and aprimaryread-write endpoint. - Branch: Isolated database environment sharing storage with parent (copy-on-write). States:
READY,ARCHIVED. - Endpoint (called Compute in UI): Compute resource powering a branch. Types:
ENDPOINT_TYPE_READ_WRITE,ENDPOINT_TYPE_READ_ONLY. - Database: Standard Postgres database within a branch. Default:
databricks_postgres. - Role: Postgres role within a branch.
Resource Name Formats
| Resource | Format |
|---|---|
| Project | projects/{project_id} |
| Branch | projects/{project_id}/branches/{branch_id} |
| Endpoint | projects/{project_id}/branches/{branch_id}/endpoints/{endpoint_id} |
| Database | projects/{project_id}/branches/{branch_id}/databases/{database_id} |
All IDs: 1-63 characters, start with lowercase letter, lowercase letters/numbers/hyphens only (RFC 1123).
CLI Discovery -- ALWAYS Do This First
Note: "Lakebase" is the product name; the CLI command group is
postgres. All commands usedatabricks postgres ....
Do NOT guess command syntax. Discover available commands dynamically:
databricks postgres -h # List all subcommands
databricks postgres <subcommand> -h # Flags, args, JSON fields
Create a Project
First decide: reuse or create. When building or attaching to an app, ask the user whether to reuse an existing project/branch/database — list them with
databricks postgres list-projects(thenlist-branches/list-databases), let the user pick, and confirm which schema the app will own — or create a new project. Only skip listing and create directly when the user explicitly asked for a brand-new project.
databricks postgres create-project <PROJECT_ID> \
--json '{"spec": {"display_name": "<DISPLAY_NAME>"}}' \
--profile <PROFILE>
Auto-creates: production branch + primary read-write endpoint with scale-to-zero on (24 h timeout). The documented default size is 8–16 CU (autoscaling), but a project created with the CLI can get a 1 CU min/max endpoint instead. Check the size with get-endpoint and resize with update-endpoint if the workload needs more. Long-running operation; CLI waits by default. Use --no-wait to return immediately.
After creation, verify:
databricks postgres list-branches projects/<PROJECT_ID> --profile <PROFILE>
databricks postgres list-endpoints projects/<PROJECT_ID>/branches/<BRANCH_ID> --profile <PROFILE>
databricks postgres list-databases projects/<PROJECT_ID>/branches/<BRANCH_ID> --profile <PROFILE>
Extract connection values from JSON output:
| Value | JSON path | Used for |
|---|---|---|
| Endpoint host | status.hosts.host | PGHOST, lakebase.postgres.host |
| Endpoint resource path | name | LAKEBASE_ENDPOINT, lakebase.postgres.endpointPath |
| Database resource path | name | lakebase.postgres.database |
| PostgreSQL database name | status.postgres_database | PGDATABASE, lakebase.postgres.databaseName |
Updating a Project
databricks postgres update-project projects/<PROJECT_ID> spec.display_name \
--json '{"spec": {"display_name": "My Updated Application"}}' \
--profile <PROFILE>
Deleting a Project
WARNING: Permanent -- deletes all branches, computes, databases, roles, and data. Do not delete without explicit user permission.
databricks postgres delete-project projects/<PROJECT_ID> --profile <PROFILE>
Autoscaling
Endpoints use compute units (CU) (~2 GB RAM per CU). Range: 0.5--64 CU (autoscaling or fixed), 65--112 CU (fixed only). Scale-to-zero enabled by default on every branch, including production (24 h timeout; computes ≤ 32 CU only).
See computes-and-scaling.md for sizing tables, endpoint CRUD, and configuration details.
Branches
Branches are copy-on-write snapshots. Use for testing schema migrations, trying queries, or previewing data changes without affecting production.
databricks postgres create-branch projects/<PROJECT_ID> <BRANCH_ID> \
--json '{"spec": {"source_branch": "projects/<PROJECT_ID>/branches/<SOURCE>", "no_expiry": true}}' \
--profile <PROFILE>
Branches require an expiration policy: "no_expiry": true for permanent, or "ttl": "<seconds>s" (max 30 days).
Limits: 500 branches per project. Per-branch storage quota is adjustable on request (contact Databricks Support). 500 Postgres roles per branch. 1,000 projects per workspace. For apps with many end users, do not create one role per user; see "Many Users: Roles and RLS at Scale" in connectivity.md.
| Use Case | TTL |
|---|---|
| CI/CD environments | 2--4 hours ("ttl": "14400s") |
| Demos | 24--48 hours ("ttl": "172800s") |
| Feature development | 1--7 days ("ttl": "604800s") |
| Long-term testing | Up to 30 days ("ttl": "2592000s") |
Point-in-time branching: Create from a past state (within restore window) for recovery. The time specification fields live inside the --json spec (spec.source_branch_time, spec.source_branch_lsn), not as top-level create-branch flags.
Reset: Replaces branch data and schema with the latest from its parent. Local changes are lost. Root branches and branches with children cannot be reset; protected branches (spec.is_protected) cannot be reset either.
Reset is UI-only — there is no databricks postgres reset-branch CLI command and no API/SDK method for it. In the Lakebase App, go to the project's Branches page, click the menu next to the branch, and select Reset from parent.
CLI workarounds:
- Delete + recreate:
delete-branch, thencreate-branchwithspec.source_branchset to the parent (unprotect viaupdate-branchfirst if protected). - Point-in-time:
create-branchwithspec.source_branch_timeorspec.source_branch_lsn. Timestamp-based reset is not supported; use PITR branching instead.
Delete: Protected branches must be unprotected first (update-branch to set spec.is_protected to false). Cannot delete branches with children. Never delete the production branch.
What's Next
Build a Databricks App
After creating a project, scaffold a connected Databricks App:
# 1. Get branch name
databricks postgres list-branches projects/<PROJECT_ID> --profile <PROFILE>
# 2. Get database name
databricks postgres list-databases projects/<PROJECT_ID>/branches/<BRANCH_ID> --profile <PROFILE>
# 3. Scaffold with lakebase feature
databricks apps init --name <APP_NAME> --features lakebase \
--set "lakebase.postgres.branch=<BRANCH_NAME>" \
--set "lakebase.postgres.database=<DATABASE_NAME>" \
--run none --profile <PROFILE>
For the full app workflow, use the databricks-apps skill.
Attach Lakebase to an existing app
apps init --features lakebase (above) wires the database at scaffold time. To
attach a project to an existing app, update its resources.
Use the postgres resource key — its fields are branch + database
(full resource paths from the table above). The legacy database key
(instance_name + database_name) is deprecated; using it fails with
Database instance <name> does not exist. Get the exact paths from
list-branches / list-databases (the DB name is often hyphenated, e.g.
databricks-postgres).
Update the app's resources with databricks apps create-update — the method to use for any app (the older databricks apps update is legacy and can't change resources for an app in a space). update_mask=resources replaces the whole resources array, so read the app's current resources and merge the new one in (or you'll detach the rest). Pass everything in --json; only APP_NAME is positional:
databricks apps create-update <APP_NAME> --json @update.json --profile <PROFILE> # waits for completion; --no-wait to return early
{
"update_mask": "resources",
"app": {
"resources": [
{
"name": "postgres",
"postgres": {
"branch": "projects/<PROJECT_ID>/branches/<BRANCH_ID>",
"database": "projects/<PROJECT_ID>/branches/<BRANCH_ID>/databases/<DATABASE_ID>",
"permission": "CAN_CONNECT_AND_CREATE"
}
}
]
}
}
Confirm the branch/database with the user — don't default to production silently. The app's service principal must be able to create and own the schema(s) it uses there, so avoid a branch/database where those schema names are already owned by a user (the SP will hit permission denied … 42501) — a fresh/dedicated branch, or a new app-owned schema, is cleanest. See Schema Permissions for Deployed Apps below for the full ownership model.
Schema Permissions for Deployed Apps
The app's Service Principal has CAN_CONNECT_AND_CREATE -- it can create new objects but cannot access existing schemas. The SP must create the schema to become its owner.
ALWAYS deploy the app before running it locally. This is the #1 source of Lakebase permission errors.
Correct workflow:
- Deploy first:
databricks apps deploy <APP_NAME> --profile <PROFILE> - Grant local access (if needed): give your own identity a role with access to the app's schema only (SQL below). Do not assign
databricks_superuserfor this. Project creators already have access. - Develop locally: your role can use only the schema and tables you granted
-- Run as the project owner. If a GRANT fails with "permission denied", run the GRANTs as the schema owner (the app's SP).
CREATE EXTENSION IF NOT EXISTS databricks_auth;
SELECT databricks_create_role('<YOUR_EMAIL>', 'USER');
GRANT USAGE ON SCHEMA <APP_SCHEMA> TO "<YOUR_EMAIL>";
GRANT SELECT, INSERT, UPDATE, DELETE ON <APP_SCHEMA>.<TABLE> TO "<YOUR_EMAIL>"; -- one line per table the app uses
GRANT USAGE ON SEQUENCE <SEQUENCE> TO "<YOUR_EMAIL>"; -- per SERIAL/identity column you insert into; name: SELECT pg_get_serial_sequence('<APP_SCHEMA>.<TABLE>', '<COLUMN>')
- This role cannot create or change schemas or tables. The app's startup DDL fails with
permission denied, evenCREATE SCHEMA IF NOT EXISTSandCREATE TABLE IF NOT EXISTSon objects that already exist. The deployed app's SP creates and migrates the schema. Locally, skip the DDL when your role lacksCREATEon the schema (has_schema_privilege('<APP_SCHEMA>', 'CREATE')). - A table added later needs its own GRANT.
The project owner and
databricks_superuserbypass RLS, even withFORCE ROW LEVEL SECURITY(both haveBYPASSRLS). A query as either one returns every row, so it cannot show that a policy works. Test isolation as a role withoutBYPASSRLS:GRANT <ROLE> TO CURRENT_USER WITH SET TRUE;, thenSET ROLE <ROLE>;, then run the app's queries.
If you already ran locally first and hit permission denied: the schema is owned by your credentials, not the SP. Do NOT drop the schema without asking the user -- dropping it deletes all data.
Ask the user to choose:
- (A) Drop and redeploy:
databricks psql --project <PROJECT_ID> -- -c "DROP SCHEMA IF EXISTS <SCHEMA_NAME> CASCADE;", thendatabricks apps deployfrom the app directory. The SP recreates the schema on startup. - (B) Export first, then drop and redeploy: export via
pg_dump(use connection details fromdatabricks postgres get-endpoint; see Other Workflows below for HOST and TOKEN) or copy tables to a temp schema usingdatabricks psql --project <PROJECT_ID>, then do option A. After the SP recreates the schema on redeploy, restore withpg_restoreor re-INSERT from the temp schema.
Other Workflows
# Connect a Postgres client -- get connection string
databricks postgres get-endpoint projects/<PROJECT_ID>/branches/<BRANCH_ID>/endpoints/<ENDPOINT_ID> --profile <PROFILE>
# Manage roles
databricks postgres create-role -h
# Add a read replica
databricks postgres create-endpoint projects/<PROJECT_ID>/branches/<BRANCH_ID> <ENDPOINT_ID> \
--json '{"spec": {"type": "ENDPOINT_TYPE_READ_ONLY"}}' --profile <PROFILE>
Run SQL against Lakebase (GRANT, CREATE INDEX, etc.):
# 1. Get endpoint host
databricks postgres get-endpoint projects/<PROJECT_ID>/branches/<BRANCH_ID>/endpoints/<ENDPOINT_ID> --profile <PROFILE>
# 2. Generate OAuth token
databricks postgres generate-database-credential \
projects/<PROJECT_ID>/branches/<BRANCH_ID>/endpoints/<ENDPOINT_ID> \
--profile <PROFILE>
# 3. Connect (use token from step 2 as password, host from step 1)
PGPASSWORD='<TOKEN>' psql "host=<HOST> user=<USERNAME> dbname=databricks_postgres sslmode=require"
Note:
generate-database-credentialrequires the endpoint resource path (.../endpoints/<ENDPOINT_ID>), not a database or branch path.
Scriptable version (single copy-paste, useful for agents):
EP=projects/<PROJECT_ID>/branches/<BRANCH_ID>/endpoints/<ENDPOINT_ID>
# get-endpoint JSON shape: {"status": {"hosts": {"host": "<HOSTNAME>"}, ...}, ...}
HOST=$(databricks postgres get-endpoint $EP --profile <PROFILE> -o json \
| python3 -c "import json,sys; print(json.load(sys.stdin)['status']['hosts']['host'])")
TOKEN=$(databricks postgres generate-database-credential $EP --profile <PROFILE> -o json \
| python3 -c "import json,sys; print(json.load(sys.stdin)['token'])")
PGPASSWORD="$TOKEN" psql "host=$HOST user=<USERNAME> dbname=databricks_postgres sslmode=require"
Grant app SP access to synced tables (run after sync is ONLINE and the app is deployed, as the identity that created the synced table: it holds SELECT ... WITH GRANT OPTION, while databricks_writer_<n> owns the table):
-- <SCHEMA>.<SYNCED_TABLE>: the last two parts of the synced table ID <UC_CATALOG>.<SCHEMA>.<TABLE>
GRANT USAGE ON SCHEMA <SCHEMA> TO "<SP_CLIENT_ID>";
GRANT SELECT ON <SCHEMA>.<SYNCED_TABLE> TO "<SP_CLIENT_ID>"; -- one line per synced table; never ALL TABLES IN SCHEMA public (it also covers Lakebase internal objects)
Run the GRANT SELECT again after each new synced table reaches ONLINE. ALTER DEFAULT PRIVILEGES does not help here: it covers only tables that its own role creates later, and synced tables are owned by databricks_writer_<dbid>.
For least-privilege, consider syncing into a dedicated schema instead of public so the grant is scoped to synced data only.
Get SP client ID: databricks apps get <APP_NAME> --profile <PROFILE> → service_principal_client_id field.
Data API: PostgREST-compatible HTTP CRUD on Postgres tables. See connectivity.md. Synced Tables: Sync Delta tables into Lakebase. See synced-tables.md.
PostgreSQL Extensions
Lakebase supports PostgreSQL extensions (e.g., pgvector for vector embeddings, pg_stat_statements for query statistics). See the full list of supported extensions.
-- List available extensions
SELECT * FROM pg_available_extensions ORDER BY name;
-- Install an extension
CREATE EXTENSION IF NOT EXISTS <extension_name>;
For vector embeddings with pgvector, see pgvector.md.
Troubleshooting
| Error | Solution |
|---|---|
cannot configure default credentials | Use --profile flag or authenticate first |
PERMISSION_DENIED | Check workspace permissions |
permission denied for schema | Schema owned by another role. If app not yet deployed: deploy first so the SP creates and owns the schema. If deployed but hitting this error (dev ran locally first): warn user about data loss, offer to export first (pg_dump with connection details from databricks postgres get-endpoint, or temp schema copy via databricks psql), then DROP SCHEMA IF EXISTS <SCHEMA_NAME> CASCADE + redeploy. |
| Protected branch won't delete | update-branch to set spec.is_protected to false first |
| Long-running operation timeout | Use --no-wait and poll with get-operation |
| New connections fail ~1 h after app start | OAuth token was fetched once at startup. Expiry is checked only at login, so open connections keep working; refresh the token before it expires and pass the current one to each new connection (see token refresh) |
| Connection refused after scale-to-zero | Compute is waking up; retry with a short backoff (see scale-to-zero) |
| Branch deletion blocked | Delete child branches first |
| Autoscaling range too wide | Max - Min cannot exceed 16 CU |
| SSL required error | Always use sslmode=require |
| Update mask required | All update-* operations require specifying fields (see -h) |
| Connection closed by server | Connections idle for 24 h are closed, and connections alive for more than 3 days may be closed regardless of activity. Use pool health checks and a max lifetime; implement retry. |
| DNS resolution fails (macOS) | Python socket.getaddrinfo() fails with long hostnames. Use dig to resolve IP, pass via hostaddr param alongside host (for TLS SNI). See connectivity.md. |
storage_catalog pipeline failure | new_pipeline_spec.storage_catalog must be a regular UC catalog, not the Lakebase catalog. DLT cannot write event logs to Postgres-backed schemas. |
| Synced table CDF error | Triggered/Continuous need a change data feed. Enable CDF on the source (ALTER TABLE ... SET TBLPROPERTIES (delta.enableChangeDataFeed = true)), use a source that Automatic change data feed supports, or use Snapshot mode. |
| Sync permissions error | Ensure USE CATALOG/USE SCHEMA on source table and CREATE TABLE in storage catalog |
| Synced table null bytes | Null bytes (0x00) in STRING/ARRAY/MAP/STRUCT columns cause sync failures. Sanitize source data: REPLACE(col, CAST(CHAR(0) AS STRING), '') |
| Synced table data modified | Only read queries, indexes, and DROP TABLE allowed on synced tables in Postgres. Modifications break sync pipeline. |
DABs synced_database_tables | Deprecated — maps to a legacy API and fails on current Lakebase. Use the postgres_synced_tables bundle resource (Beta, CLI v1.0.0+) or the databricks postgres create-synced-table CLI instead. |
SDK and Version Requirements
| Component | Minimum Version |
|---|---|
| Databricks CLI | >= v0.294.0 |
| Databricks SDK for Python | >= 0.81.0 (for w.postgres module) |
| psycopg | 2.x or 3.x (3.x recommended for async/pooling) |
| Postgres | 16, 17, or 18 (default: PG 17) |
Lakeflow Connect: SaaS connectors
FIRST: use the parent databricks-core skill for CLI install, authentication, and profile selection.
| Source | Skill |
|---|---|
| A SaaS application, a RabbitMQ queue, or a Kafka topic with the managed connector (Beta) | this skill |
| A Kafka topic whose messages need more than the key and value formats the Kafka pipeline page lists | read_kafka in databricks-pipelines, or databricks-spark-structured-streaming |
| A database (SQL Server, PostgreSQL, MySQL, Oracle, query-based) | databricks-lakeflow-connect |
| An app or device that pushes events | databricks-zerobus-ingest |
| Files on cloud object storage | Auto Loader, databricks-pipelines |
| Files on an SFTP server | SFTP file connector on Auto Loader: file-connectors.md |
| A stream with no managed connector (Kinesis, Pub/Sub, Pulsar) | databricks-spark-structured-streaming |
| A source with no managed connector | a custom connector (Beta) or a community connector |
Find the connector's pages
Connectors and options change faster than this skill: read the connector's docs pages instead of answering from memory. When a page and this skill disagree, the page wins.
- Connector keys: the links on SaaS connectors, streaming connectors (Kafka, RabbitMQ) and file connectors. Say a connector isn't available only when none of them lists it.
- Keys on those pages as of 2026-11 (newer ones may be there): 1password-event-logs, aha, akamai-waf, amplitude, anaplan, anthropic, anysphere-audit-logs, anysphere-organization, asana, atlassian-audit-logs, azure-devops, box-audit-logs, celigo, confluence, cribl, crowdstrike-falcon-event-stream, d365, github, gitlab, glean, gmail, google-ads, google-analytics, google-analytics-aggregated, google-drive, google-search-console, google-workspace, hubspot, jira, kafka, linkedin-ads, marketo, meta-ads, microsoft-365, microsoft-entra-id, microsoft-graph-logs, monday-com, netskope-logs, netsuite, notion, okta, okta-system-logs, onedrive, openai, oracle-eloqua, outlook, pagerduty, paypal, pendo, productboard, quickbooks, rabbitmq, reddit-ads, salesforce, sendgrid, servicenow, sfmc, sharepoint, shopify, smartsheet, square, strac, surveymonkey, tiktok-ads, veeva-vault, verkada, vimeo, wiz-audit-logs, workday-activity-logging, workday-hcm, workday-reports, workiva, xero, zendesk-support, zip, zoho-books, zoho-crm.
- Pages:
https://docs.databricks.com/ingestion/lakeflow-connect/<key>(or<key>-overview), then-source-setup,-connection,-pipeline,-reference,-limits,-troubleshoot(not every connector has each). - Read pages with the web-fetch tool or the databricks-docs skill. If neither is available, give the user the page links and ask what they show; don't fetch another way.
- Read
-referencebefore naming a source table or saying which tables are incremental;-limitsbefore sizing. - Notes to read before the steps: Google, Meta, LinkedIn, Reddit, TikTok Ads → ads.md; SharePoint, Google Drive, OneDrive, SFTP → file-connectors.md; Kafka, RabbitMQ → streaming.md; the rest → connector-notes.md.
Workflow
Ask for every missing input in one message (connector, objects, destination catalog and schema, schedule), list what you'll create, and wait for one "yes".
- Previews toggle. When the page's Beta note mentions the Previews page, a workspace admin turns that preview on first. Beta has no production SLA: say so.
- Source setup. Follow
-source-setupon the SaaS side (app registration, API user, scopes). Connectors with several auth methods have one setup page each. - Connection. Created in Catalog Explorer (the
-connectionpage has the exact path), often with a browser sign-in. Ask the user to create it from the-connectionpage and give you its name; don't guessconnection_typeor option names fordatabricks connections create. Grants: databricks-unity-catalog. - Pipeline. Copy the object shape from the
-pipelinepage examples.source_schemais usuallydefault, the account ID for ads,objectsfor Salesforce; never a placeholder. - Run and schedule. Start one update and check it, then schedule a Jobs
pipeline_task(databricks-pipelines, databricks-jobs). Anything kept: a bundlepipelinesresource with thisingestion_definitionplus a job with apipeline_task(databricks-dabs). Kafka and RabbitMQ run continuously, with no job.
{
"name": "jira_ingest",
"catalog": "main",
"schema": "jira_raw",
"ingestion_definition": {
"connection_name": "<jira-connection>",
"objects": [
{"table": {"source_schema": "default", "source_table": "issues",
"destination_catalog": "main", "destination_schema": "jira_raw"}}
]
}
}
connection_nameis the UC connection's name, not its ID.objects[].tableper table,objects[].schemafor every table of a source schema, orobjects[].reportwithsource_url(Workday Reports).connector_optionscarries connector settings (lookback windows, report levels, URLs); the keys don't follow the connector key (Google Drive isgdrive_options): take them fromdatabricks bundle schema(pipelines.ConnectorOptions).table_configurationcarries per-table settings such asscd_type: SCD_TYPE_2, where the overview's feature table allows it.channel: PREVIEWonly where the docs examples set it: SharePoint (GA), Google Drive, OneDrive, Kafka, RabbitMQ.
Important
- An ingestion pipeline has
ingestion_definition, neverlibraries. - Ask before reusing a destination schema: a source table whose name matches an existing destination table fails the update.
- Two tables with the same name can't share a pipeline, even from different source schemas: set
destination_tableon one, or split. A renamed table, or several destination catalogs or schemas, make the pipeline API-only. - A column selected after the pipeline started isn't backfilled. A full refresh backfills it but clears the table and its state and reloads from the source: SCD type 2 history, rows deleted at the source and data the source no longer keeps are lost. Say so and ask before running one.
- A table deleted at the source keeps its destination table.
Troubleshooting
Read the pipeline's event log (databricks-pipelines), then the connector's -troubleshoot page.
| Symptom | Cause | Fix |
|---|---|---|
DUPLICATE_KEY_VIOLATION | The key the source reports isn't unique per row (NetSuite's line-level tables report transactionid) | Set table_configuration.primary_keys to a key that is unique per row, as the connector's -pipeline page names it (NetSuite: transactionid and line), then full refresh that table (ask first) |
| A working connection fails auth | Refresh token revoked or expired | Reauthorize the connection in Catalog Explorer |
| Create fails or the connector isn't found | Previews toggle off, or channel: PREVIEW missing | Check the page's Beta note and examples |
| Every update re-reads a whole table | No cursor (entity tables, formula fields, report snapshots) | Expected; Salesforce formula fields can be ingested incrementally (Beta) in a new pipeline, API only: Ingest formula fields incrementally |
| Too many tables | Per-pipeline cap on -limits (250 where stated, NetSuite 200, Outlook 1) | Split the pipeline |
Reference Files
- ads.md: ads entity vs report tables, account-ID schemas, options.
- file-connectors.md: SharePoint, Google Drive, OneDrive, SFTP.
- streaming.md: Kafka and RabbitMQ.
- connector-notes.md: traps for other connectors, A to Z.
Lakeflow Connect: databases
Managed ingestion pipelines from databases into Unity Catalog Delta tables (SaaS apps and Kafka / RabbitMQ topics: databricks-lakeflow-connect-saas).
The docs pages win. Release stage, prerequisites, grants, limits, error text and field names change between releases: read the source's pages below before you build, and the workspace's error before this skill. This skill holds the order of steps, the gates, and the traps the pages don't state.
Is Lakeflow Connect the right tool?
| If the source is... | Use | Skill |
|---|---|---|
| A database with a managed connector | Lakeflow Connect | this skill |
| A SaaS app (Salesforce, Workday, ServiceNow, SharePoint, ...) or a Kafka / RabbitMQ topic | Lakeflow Connect | databricks-lakeflow-connect-saas |
| Files on cloud object storage | Auto Loader | databricks-pipelines |
| A source to query in place, no copy | Lakehouse Federation | n/a |
| An app or device that pushes events | Zerobus | databricks-zerobus-ingest |
Modes per source
Pages are https://docs.databricks.com/ingestion/lakeflow-connect/<source>-<page>. For a mode marked "-", look for its page first (<source>-integrated-pipeline for integrated CDC, <source>-pipeline for standard CDC): if it exists, follow it; if it isn't found, tell the user Lakeflow Connect's docs have no page for
Source (<source>) | Integrated CDC | Standard CDC (gateway) | Query-based | Setup, limits, errors |
|---|---|---|---|---|
SQL Server (sql-server) | integrated-pipeline | pipeline | yes | source-setup, limits, troubleshoot |
MySQL (mysql) | integrated-pipeline | pipeline | yes | source-setup, limits, troubleshoot |
PostgreSQL (postgresql) | - | pipeline | yes | source-setup, limits, troubleshoot |
Oracle (oracle) | integrated-pipeline | - | yes | integrated-setup, limits, troubleshoot |
| Teradata, MariaDB, any Federation foreign catalog | - | - | yes | query-based overview, limits |
Which mode: CDC without a kind named: ask which (integrated, one pipeline; standard, a gateway that runs and bills continuously), unless the source has one. "Standard", "two-pipeline" or "gateway" → standard. Every change or low source load → CDC (hard deletes with query-based: its limits page); otherwise, or when CDC can't be turned on at the source → query-based. A mode marked "-" with no page found (or the docs not reachable): tell the user as in Modes per source, offer the modes it has, and don't author the requested one, even to try it. Never switch to standard silently: its gateway runs and bills continuously.
What marks each mode in the pipeline
Field names and stages: databricks bundle schema (pipelines.IngestionPipelineDefinition). An ingestion pipeline has ingestion_definition, never libraries; CREATE TABLE ... FROM CONNECTION is Federation, not ingestion.
| Mode | Pipelines | Marking fields |
|---|---|---|
| Integrated CDC | one | connection_name + connector_type: CDC; data_staging_options (catalog, schema); every field the source's page says a create request must include (Oracle and MySQL: channel: PREVIEW) |
| Standard CDC | gateway + ingestion | gateway: gateway_definition (connection_name, gateway_storage_catalog, _schema, _name); ingestion: ingestion_gateway_id: ${resources.pipelines.<gateway>.id}; source_type and the source's settings where its page's example has them (PostgreSQL: source_type: POSTGRESQL, source_configurations[].catalog.postgres.slot_config) |
| Query-based, connection | one | connection_name, no connector_type; per table table_configuration.query_based_connector_config.cursor_columns (a list), primary_keys, scd_type |
| Query-based, foreign catalog | one | ingest_from_uc_foreign_catalog: true, no connection_name; source_catalog = the foreign catalog; primary_keys required unless APPEND_ONLY |
Every object: table with source_catalog / source_schema / source_table, destination_catalog / destination_schema (destination_table only to rename), or schema for a whole source schema. Per source: sources.md.
Compute
Before creating, find what each pipeline would run on with serverless unset: the mode's pipeline page, and pipelines of the same mode already in this workspace (databricks pipelines list-pipelines; update.config.serverless of their latest update in databricks pipelines get-update <id> <update-id> -o json: true serverless, false classic). Compare that with the compute the user asked for: set serverless (and a clusters block for classic, as the mode's page shows) only when they differ, otherwise leave it unset. After the first update, read the same field to confirm. If the page says a mode or one of its pipelines (a gateway, an ingestion pipeline) can't run on the compute asked for, say so and offer what the page allows.
Details and the two compute failures: modes.md.
Before creating anything
Ask for every missing input in one message (source, connection, tables, destination catalog and schema, mode, compute, schedule), list what you'll create, and wait for one "yes"; then run the setup without asking again. Never create a stand-in source without the user's go-ahead. Check:
- Target catalog and grants: as the mode's pipeline page lists.
- A UC
CONNECTIONwith the source credentials: password assecret('<scope>', '<key>')(CREATE CONNECTION: use the secret function for any credential). After a rotation, runALTER CONNECTION <name> OPTIONS (..., password secret('<scope>', '<key>'))with the options from theCREATE(DESCRIBE CONNECTIONlists only some), then test the connection with a read. Write a request body with a password to a file only you can read, never on a command line. - Network from the compute that reads the source (the integrated pipeline, the gateway, or the query-based pipeline, on the compute chosen above).
- Source setup from the source's
source-setuppage, and its traps in sources.md. Changes to the customer's database (CDC on a table, a slot): give the SQL to the user or their DBA; ask first. - Query-based: the pre-flight checks (key unique, cursor bumped on update, no NULL cursors). Report, and let the user skip them for a snapshot or test.
- Names: ask. Demos and tests without names: one
<base>_<epoch>for pipeline, schema, job and bundle (standard CDC:<base>_gw_<epoch>and<base>_ap_<epoch>; staging<base>_<epoch>_staging).mode: developmentprefixes names, so slot and schema names taken from them stop matching.
Run, schedule, clean up
databricks bundle deploy, thendatabricks bundle run <key> --no-wait(orpipelines start-update <id>), thenget-updateuntilCOMPLETEDorFAILED; integrated CDC updates run long by design (smart closure).- Schedule with a Jobs
pipeline_task; whether the mode can also run continuously, the mode's pipeline page says. - After the first load and every full refresh, compare source and destination row counts; a
COMPLETEDstatus or low lag isn't proof rows landed. Rows per table per run: rows-per-run.md. - Before a full refresh, read what the mode's page says it does to the table and to change-feed readers; warn and ask first.
- Clean up only with the user's OK: list the pipelines, schemas and PostgreSQL slot you'd remove, and what deleting a pipeline does to its tables (its page). Give the slot SQL to the user or their DBA. Don't stop a gateway the user keeps.
When something fails
Read the full message (CLI error, deploy output, ERROR/WARN events with details), then the source's troubleshoot page, then troubleshooting.md and sources.md. Then decide: a setting to change (take the value the message names), a missing grant (report the privilege and who grants it), a feature not enabled ("Ingestion source 'bundle destroy for what a failed deploy left.
Reference files
| File | When |
|---|---|
| modes.md | Integrated CDC, standard CDC, query-based: how each behaves, one example each, compute failures |
| sources.md | SQL Server, PostgreSQL, MySQL, Oracle: names per source, prerequisite traps, errors from runs |
| troubleshooting.md | Event log, cross-pipeline errors, table limits, runtime per update |
| rows-per-run.md | Rows per table per run (scripts/rows_per_run.sql) |
Related skills
databricks-pipelines (runtime, event log), databricks-dabs (bundles), databricks-jobs (pipeline_task), databricks-unity-catalog (catalogs, connections), databricks-lakebase (Lakebase projects and roles), databricks-zerobus-ingest (push).
Databricks Lakehouse MCP
The hosted system.ai.lakehouse service provides notebook tools and the detailed
instructions for using them. This local skill handles discovery and connection;
load operational guidance from the connected service for the user's task.
Connect or recover
First use the client's tool discovery to look for configured or deferred
Lakehouse MCP tools. If they are available, proceed to skill retrieval.
Otherwise, use the setup instructions shown for system.ai.lakehouse in the
user's Databricks workspace to configure and authenticate the connection in
Claude Code or Codex. Use the service's current setup instructions for connection
details; do not guess an endpoint or installer command.
If system.ai.lakehouse or its setup instructions are unavailable, or preview
access is denied, direct the user to their Databricks account representative to
establish availability and request access. Installing this skill does not grant
MCP access. Missing tools alone do not show whether the connection is
unconfigured or access is unavailable.
For an authentication error, follow the client's sign-in recovery. For an access denial, use the relevant access-request path. For a network or service error, guide retry or diagnosis. If the cause is unclear, keep it unknown. After explaining a blocker, avoid repeating the same setup prompt on later turns; continue through an existing supported path when appropriate for the request.
Use databricks-core when CLI authentication or profile selection is needed.
Retrieve the task guidance
- Discover the connected server's advertised skill tools. Call
list_skills(query="<short task keyword>")to search skill IDs, names, and descriptions by case-insensitive substring. Omitqueryto browse the available guides, including when a search returns no matches. Follow any returnedpage_tokento see more results. - Load the relevant guide with
load_skill(skill="<returned skill id>"). This returns the fullSKILL.mdinstructions and supporting-file metadata. - Read the supporting files needed for the task with
read_skill_file(skill="<returned skill id>", path="<returned file path>"). Use the exact paths returned byload_skillorlist_skill_files. To browse more files, calllist_skill_files(skill="<returned skill id>"); an optionalprefix, such as"references/", narrows the list. Follow any returned next-call instructions to continue browsing with the same filters andpage_token. - Follow the guide and current tool descriptions to perform the task. Load more guides when the task requires them, and reuse applicable guidance while it remains in context. Refresh the relevant guidance after reconnecting or detecting a change in the service's capabilities.
Skill retrieval is a useful connection check; no separate healthcheck is needed. It verifies access to that read, not permission to edit a notebook or readiness to run compute. Handle those checks and failures through the relevant operations. If the server does not advertise skill retrieval, report the missing capability and use the service's current setup/recovery guidance.
Unity Catalog Metric Views
Define reusable, governed business metrics in YAML that separate measure definitions from dimension groupings for flexible querying.
When to Use
Use this skill when:
- Defining standardized business metrics (revenue, order counts, conversion rates)
- Building KPI layers shared across dashboards, Genie, and SQL queries
- Creating metrics with complex aggregations (ratios, distinct counts, filtered measures)
- Defining window measures (moving averages, running totals, period-over-period, YTD)
- Modeling star or snowflake schemas with joins in metric definitions
- Enabling materialization for pre-computed metric aggregations
First: can this even be a metric view? (decision gate)
Before authoring, confirm the measure re-aggregates at query time. Additive SUM/COUNT, a COUNT(DISTINCT) that recomputes per cell, MEASURE() ratios (divide last), and supported window measures (trailing / cumulative / period-over-period — Pattern 8) all re-aggregate correctly and are valid metric-view measures; this list is not exhaustive. The gate below is about disqualifying shapes — if any of the following hold, it is not a metric view: author a governed Unity Catalog SQL function instead (build it with databricks-dbsql) and expose it to Genie as a trusted registered SQL function (Design Priorities surface #11), rather than forcing it into a view or a pre-aggregated helper view:
- Non-additive at the queried grain — summing it across a dimension double-counts. Distinct-entity counts are the classic trap: an entity appearing under two attribute values is one entity, not two, so a per-cell distinct count must not be summed across an arbitrary multi-attribute cross.
- Computed per-entity, then aggregated, where the per-entity step isn't a sum/count/ratio-of-sums — e.g. a per-entity min/max across two periods, or a threshold on a per-entity distinct count.
- Selection-dependent — the result redistributes with the user's filter (an "all others" bucket that shifts every cell).
- Iterative — repeats until convergence (e.g. panel reweighting).
- Returns a bundle of tables, not a single measure (e.g. a source×destination matrix plus per-row nets).
A measure can be mostly a metric view with one part hoisted into a function (penetration = additive buyers ÷ base, divide last — but a short-window correction, or an arbitrary multi-attribute cross / per-entity threshold, goes in a function). Pre-computing the awkward step into a base/helper view is only a fix for a fixed definition — a performance choice, settled once for all slices. If the measure must respond to the user's runtime selection in Genie (arbitrary cross, a parameter, or a shifting denominator), it is computed on the fly and cannot be pre-computed — it needs a parameterized governed SQL function, not a pre-aggregated view. Full rule + two worked examples (penetration, brand-switching gains & loss): metric-view-advisor.md §When a metric view is not the right tool. Before benchmarking Genie's routing to a function that reproduces an external methodology, validate it to N-decimal parity against its reference — see the databricks-genie-agents skill.
Prerequisites
- Databricks Runtime 17.2+ (for YAML version 1.1); 17.3+ for semantic metadata (
synonyms/display_name/format) - SQL warehouse with
CAN USEpermissions SELECTon source tables,CREATE TABLE+USE SCHEMAin the target schema
Metric View Lifecycle
| Task | Reference | Load when |
|---|---|---|
| Create | metric-view-advisor.md | Any creation task — the advisor handles the full workflow (profile schema, analyze sources, suggest, deploy). Load create-patterns.md alongside as the YAML spec and pattern reference. |
| YAML spec / patterns | create-patterns.md | Patterns 1–12, full YAML field reference, formatting gotchas, deployment errors, quick reference. Companion to the advisor; also load directly for pattern lookup. |
| Query | query-patterns.md | Writing SQL against a metric view — MEASURE() basics, filters, join rollups, window measures, Rules 1–3. |
| Genie integration | metric-view-advisor.md §Genie Design Rules | One-fact-source rule, base views, domain organization, naming. Agent metadata fields (comment, synonyms, display_name, format) are in create-patterns.md §YAML Field Reference. |
Typical flow: advisor → create → query/validate → Genie integration (if adding to a Genie Agent).
Source-controlled deployment with Declarative Automation Bundles
To source-control a metric view, commit its complete SQL definition and execute it through a bundle-managed SQL job. DABs do not have a native metric-view resource, but a bundle-managed SQL job can apply a committed definition:
# databricks.yml
bundle:
name: orders_metrics
variables:
catalog: { default: main }
schema: { default: default }
warehouse_id: { default: "" }
resources:
jobs:
deploy_orders_metrics:
name: deploy_orders_metrics
parameters:
- name: catalog
default: ${var.catalog}
- name: schema
default: ${var.schema}
tasks:
- task_key: create_metric_view
sql_task:
warehouse_id: ${var.warehouse_id}
file:
path: ../src/orders_metrics.metric_view.sql
Deploy and run:
databricks bundle deploy --target <TARGET> --profile <PROFILE>
databricks bundle run deploy_orders_metrics --target <TARGET> --profile <PROFILE>
See the official metric view bundle example.
Related Skills
- databricks-genie-agents — create, manage, and validate Genie Agents that consume the metric views built here. Metric-view design rules for Genie are in the advisor §Genie Design Rules; query rules are in query-patterns.md.
- databricks-aibi-dashboards — build AI/BI dashboards on top of metric views.
- databricks-data-discovery — explore data before creating metric views; answer questions across your workspace.
- databricks-dabs — source-control and deploy metric view SQL definitions via bundle-managed jobs.
Resources
ML Training on Databricks
FIRST: Use the parent databricks-core skill for CLI basics, authentication, and profile selection.
Train with MLflow → register to Unity Catalog → consume the same artifact as either a batch Spark UDF over Delta or (when low-latency is required) a real-time serving endpoint.
Always train on Databricks (serverless job or notebook), never in the local Python process the agent is running in. Local training has no access to the silver tables, no MLflow tracking server, no UC registry path, and dies if the chat session drops — submit
databricks jobs submit --no-wait(see "Train + deploy as a serverless job" below). Only fall back to local execution if the user explicitly asks for it.
If you need to deploy a real time model serving endpoint after the model is registered (creating endpoints, traffic config, version-swapping, querying, Foundation Model API endpoints), see databricks-model-serving.
| Consumption | When | How |
|---|---|---|
| Batch UDF | Dashboards, daily/hourly scores, predictions read by Genie/Dashboards or an app (often synced to a Lakebase table) | mlflow.pyfunc.spark_udf(...) → INSERT INTO gold_predictions. If the model was logged with fe.log_model(training_set=...), use fe.score_batch() instead — see the Feature Engineering section below. |
| Real-time endpoint | Score on a user action (fraud at authorization, rec at page load) — sub-100ms | mlflow.deployments.get_deploy_client() (classical) / agents.deploy() (agents). Endpoint lifecycle: see databricks-model-serving. |
Default Canonical flow
silver_<features> + silver_<labels>
▼
notebook (as a serverless job):
├── train with mlflow.autolog (XGBoost / sklearn / etc.)
├── mlflow.register_model → UC: {catalog}.{schema}.{model}
├── set_registered_model_alias(name, "prod", version)
└── spark_udf(@prod) over latest features → MERGE into gold_predictions
▼
gold_<entity>_predictions ◄── dashboards, apps, Genie read this
Feature-store-backed models diverge here. If training used
fe.log_model(training_set=...), replacespark_udf(@prod)withfe.score_batch(model_uri, df=<keys_only>)— it auto-joins features via the model's registered feature lineage. See the Feature Engineering section below.
One notebook, one artifact. Re-running = retraining. Gold is where truth lives — read paths never call the model directly. Keep label-window logic (failure occurred within 7 days) in the notebook during dev; once stable, promote to a silver materialized view in SDP.
Train and register (the 90% case)
mlflow.autolog() captures params, metrics, code, and the model artifact for every run; registered_model_name=... auto-registers the best run to UC (auto-incremented version). Wrap training with Optuna so each trial is a child run and the best one is what gets registered.
Always mlflow.set_registry_uri("databricks-uc") — without it, models land in the deprecated workspace registry. The experiment's parent folder must exist — set_experiment does NOT auto-create it (fails with NOT_FOUND: Parent directory does not exist). Pre-create it once with databricks workspace mkdirs before the job runs.
# Once per project — create the parent folder for the MLflow experiment.
databricks workspace mkdirs /Users/me@example.com/turbine_project
Use the Databricks notebook source format (# Databricks notebook source header, # COMMAND ---------- separators, # MAGIC %md/%sql magics for markdown/SQL cells):
# Databricks notebook source
# MAGIC %md
# MAGIC # Turbine failure prediction
# MAGIC
# MAGIC Train an XGBoost classifier on engineered turbine telemetry features.
# MAGIC ## Data exploration
# COMMAND ----------
# (basic data exploration — class balance, schema sanity, etc.)
# COMMAND ----------
# MAGIC %md
# MAGIC ## Training the model
# COMMAND ----------
import mlflow, mlflow.xgboost, optuna
from mlflow.tracking import MlflowClient
from xgboost import XGBClassifier
from sklearn.metrics import roc_auc_score
mlflow.set_registry_uri("databricks-uc")
mlflow.set_experiment("/Users/me@example.com/turbine_project/mlflow_experiment")
CATALOG, SCHEMA, NAME = "ai_demo_gen", "wind_farm", "turbine_failure"
FULL_NAME = f"{CATALOG}.{SCHEMA}.{NAME}"
# Autolog WITHOUT registered_model_name — otherwise every Optuna trial registers a new UC
# version, and a max-by-version pick lands on the last trial to finish, not the best one.
mlflow.xgboost.autolog(log_input_examples=True)
# For imbalanced labels: stratify the split, set scale_pos_weight = neg/pos.
def objective(trial):
params = {
"n_estimators": trial.suggest_int("n_estimators", 100, 400),
"max_depth": trial.suggest_int("max_depth", 3, 10),
"learning_rate": trial.suggest_float("learning_rate", 0.01, 0.3, log=True),
}
with mlflow.start_run(nested=True):
m = XGBClassifier(**params).fit(X_train, y_train)
return roc_auc_score(y_test, m.predict_proba(X_test)[:, 1])
with mlflow.start_run(run_name="hpo") as parent:
study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=20)
# COMMAND ----------
# MAGIC %md
# MAGIC ## Retrain best params and register
# COMMAND ----------
# Retrain on the winning trial's params explicitly, then register that single model.
with mlflow.start_run(run_name="best"):
best = XGBClassifier(**study.best_params).fit(X_train, y_train)
mlflow.log_metric("val_auc", study.best_value)
info = mlflow.xgboost.log_model(best, name="model", registered_model_name=FULL_NAME)
# Stages are deprecated — UC uses movable aliases. Repoint @prod at the version we just registered.
client = MlflowClient(registry_uri="databricks-uc")
client.set_registered_model_alias(FULL_NAME, "prod", info.registered_model_version)
Framework autolog: mlflow.{sklearn,xgboost,lightgbm,pytorch,tensorflow,spark}.autolog().
Aliases, not stages: UC dropped Staging/Production. Use movable @prod/@challenger; load with models:/{full_name}@prod. Promoting a new version is one set_registered_model_alias call.
Consume: batch scoring over Delta
The cheap, default path for models NOT backed by feature tables. Load the registered model as a Spark UDF and score a Delta table; write predictions to a gold table that downstream consumers read. For feature-store-backed models (logged with fe.log_model(training_set=...)), skip spark_udf entirely and use fe.score_batch() — see the Feature Engineering section.
# COMMAND ----------
# MAGIC %md
# MAGIC ## Score and save to a gold predictions table
# COMMAND ----------
import mlflow
from pyspark.sql import functions as F
# env_manager rules:
# "local" → same runtime as training (same notebook/job). Fastest, default in dev/demo.
# "virtualenv"→ different runtime than training; rebuilds the model's env.
# "uv" → same as virtualenv but faster (MLflow ≥ 2.22).
predict = mlflow.pyfunc.spark_udf(
spark,
model_uri=f"models:/{FULL_NAME}@prod",
env_manager="local",
)
features = spark.table(f"{CATALOG}.{SCHEMA}.silver_turbine_features_latest")
feature_cols = [c for c in features.columns if c != "turbine_id"] # exclude the join key
scored = features.withColumn("risk_score", predict(*[features[c] for c in feature_cols]))
# Overwrite-per-run pattern for "latest score per entity":
scored.select("turbine_id", "risk_score", F.current_timestamp().alias("scored_at")) \
.write.mode("overwrite").saveAsTable(f"{CATALOG}.{SCHEMA}.gold_turbine_predictions")
For incremental scoring with history, MERGE into the predictions table instead of overwrite.
Real-time serving (when required)
After registering a model to UC, deploy it behind a Model Serving endpoint. The dev-side call is mlflow.deployments.get_deploy_client("databricks").create_endpoint(...) for classical ML or agents.deploy(...) for ResponsesAgents. First deploy is ~5 min for classical ML.
For endpoint create / update / version-swap, traffic config, AI Gateway, querying, the state.ready + state.config_update two-field readiness check, and Foundation Model API endpoints, see databricks-model-serving.
Train + deploy as a serverless job
Training notebooks run a few minutes (Optuna + UC register; endpoint warmup adds 5–15 min if you also deploy). Submit as a serverless one-time run so the CLI doesn't block. The notebook ends with dbutils.notebook.exit(json.dumps({...})) so the structured result (model_version, val_auc, endpoint_name) reaches .notebook_output.result.
# 1. Upload the training notebook
databricks workspace import /Workspace/Users/me@example.com/turbine_project/train \
--file ./train_notebook.py --format SOURCE --language PYTHON --overwrite
# 2. Submit as serverless one-time run (returns {"run_id": N} immediately with --no-wait)
RUN_ID=$(databricks jobs submit --no-wait --json '{
"run_name": "turbine-train-and-deploy",
"tasks": [{
"task_key": "train",
"notebook_task": {"notebook_path": "/Workspace/Users/me@example.com/turbine_project/train"},
"environment_key": "ml_env"
}],
"environments": [{
"environment_key": "ml_env",
"spec": {
"client": "4",
"dependencies": ["mlflow==3.1.0", "xgboost==2.1.3", "optuna==4.1.0", "scikit-learn==1.5.2"]
}
}]
}' | jq -r .run_id)
# 3. Poll until a terminal life_cycle_state.
for _ in $(seq 60); do
STATE=$(databricks jobs get-run "$RUN_ID" | jq -r '.state.life_cycle_state // "UNKNOWN"')
echo "$(date +%H:%M:%S) $STATE"
[[ "$STATE" =~ ^(TERMINATED|SKIPPED|INTERNAL_ERROR)$ ]] && break
sleep 30
done
[[ "$STATE" =~ ^(TERMINATED|SKIPPED|INTERNAL_ERROR)$ ]] || { databricks jobs cancel-run "$RUN_ID"; exit 1; }
# life_cycle_state TERMINATED only means "the run ended" — check result_state.
RESULT=$(databricks jobs get-run "$RUN_ID" | jq -r '.state.result_state // "UNKNOWN"')
echo "result_state=$RESULT"
[[ "$RESULT" == "SUCCESS" ]] || { echo "Run did not succeed"; exit 1; }
# 4. Pull structured output via the TASK run_id (NOT the submit run_id).
TASK_RUN_ID=$(databricks jobs get-run "$RUN_ID" | jq -r '.tasks[0].run_id')
databricks jobs get-run-output "$TASK_RUN_ID" | jq '.notebook_output.result'
# → '{"model_version":"3","val_auc":0.91,"rows_scored":124,"endpoint":"turbine-risk-endpoint"}'
Common jobs submit traps to be aware of: environments[].spec.client: "4" is required on serverless notebook tasks; use the TASK run_id (tasks[0].run_id) — NOT the submit run_id — for get-run-output; print() is unreliable on serverless one-time runs (use dbutils.notebook.exit(json.dumps(...))); jobs submit rejects tags. For the broader databricks-jobs skill, see databricks-jobs.
Custom pyfunc
When sklearn/XGBoost autolog isn't enough — custom preprocessing, multiple sub-models, external API calls, ensemble logic. See references/custom-pyfunc.md for a full worked example. Two non-obvious things:
python_model="path/to/file.py"(file path, not class instance) +mlflow.models.set_model(MyModel())at the end of that file. This is the "Models from Code" pattern — the file is logged verbatim, no pickling of the class.mlflow.models.predict(model_uri=..., input_data=..., env_manager="uv")before deploying. Catches missing deps before the endpoint does.
Custom GenAI agents
Hand-rolled ResponsesAgent (LangGraph + UC Function tools + Vector Search retrieval) — see references/genai-agents.md.
Prefer no-code authoring via databricks-genie-agents (tables + documents in UC volumes; replaces the legacy Knowledge Assistant / Supervisor Agent) unless the user explicitly needs a custom LangGraph agent.
Feature Engineering: Feature Store & Feature Views
When to reach for Feature Engineering (either flavor) instead of a plain Delta table: when the same feature must be computed identically at training and serving time (no training/serving skew), the feature is time-dependent and needs point-in-time joins against labels (no future leakage), the feature needs to be served with <10ms latency via an online store, or the feature is shared across models with lineage tracked in UC. If none apply, a plain UC table is enough — the sections below can be skipped.
Default: don't use Feature Engineering unless one of the reasons above clearly applies or the user explicitly asked for it. It adds build time and complexity (an extra Delta table layer for FeatureLookup; a materialization pipeline for Feature Views). If you're unsure, use plain UC tables and add Feature Engineering later when a concrete need surfaces.
Two flavors, one train/score path (create_training_set → fe.log_model → fe.score_batch):
FeatureLookupAPI — you own the feature tables (compute, write, refresh); bind them to a training set viaFeatureLookup+FeatureEngineeringClient. Reach for it when the features already exist as Delta tables or you want direct control over how they are computed. See references/feature-store.md.- Feature Views (declarative, GA,
databricks-feature-engineering>=0.16.0) — Databricks owns compute, materialization (Delta offline + Lakebase online), and refresh from a spec you declare (create_featureoverDeltaTableSource;RollingWindow/SlidingWindow/TumblingWindow;create_streamfor Kafka features). Reach for it when the feature is a formula you want Databricks to keep fresh. See references/feature-views.md.
Gotchas (the ones that cost time)
| Trap | Fix |
|---|---|
| Model lands in workspace registry, not UC | mlflow.set_registry_uri("databricks-uc") before logging |
| Endpoint returns PERMISSION_DENIED at first query | Pass resources=[...] to log_model (covers UC functions, VS indexes, other endpoints, Lakebase) — see references/genai-agents.md#resources-that-need-passthrough-auth for the full list |
Used transition_model_version_stage | Stages are deprecated in UC. Use client.set_registered_model_alias(name, "prod", version) |
spark_udf rebuilds a virtualenv on every call | Pass env_manager="local" when training+scoring share a runtime |
pip_requirements mismatch crashes endpoint at load | Pin exact versions; or pull live with f"mlflow=={get_distribution('mlflow').version}" |
agents.deploy() produced a weirdly-named endpoint | Pass endpoint_name=... explicitly. Auto-derived name is agents_<catalog>-<schema>-<model> |
Endpoint-lifecycle gotchas (readiness two-state, version-swap, Serving-UI SP filter) live in databricks-model-serving.
Reference files
| File | Contents |
|---|---|
| references/custom-pyfunc.md | Single end-to-end custom pyfunc example: artifacts, signature, code_paths, log → register → deploy → query. |
| references/genai-agents.md | Custom LangGraph ResponsesAgent with UC Function + Vector Search tools. create_text_output_item gotcha and the resources=[...] passthrough-auth list. For no-code agents prefer databricks-genie-agents. |
| references/feature-store.md | Feature Engineering in Unity Catalog (standard FeatureLookup API): FeatureEngineeringClient, create_table with timeseries_column, FeatureLookup with point-in-time joins, fe.log_model with lineage, fe.score_batch (replaces spark_udf for FE models), and Lakebase online store via publish_table. Requires databricks-feature-engineering>=0.16.0. |
| references/feature-views.md | Feature Views (declarative Feature Engineering, GA): create_feature over DeltaTableSource, RollingWindow/SlidingWindow/TumblingWindow aggregations, materialize_features (offline + online), streaming features off Kafka via create_stream, point-in-time training sets, and the Feature Serving Endpoint. Requires databricks-feature-engineering>=0.16.0. |
Related skills
- databricks-model-serving — serving-endpoint lifecycle (create, query, update-config, version-swap, AI Gateway, Foundation Model API endpoints).
- databricks-genie-agents: no-code agents over tables + documents (replaces legacy Knowledge Assistants / Supervisor Agents). Prefer this over hand-rolling agents.
- databricks-mlflow-evaluation — evaluate model/agent quality before promoting
@prod. - databricks-vector-search — vector indexes used as retrieval tools in agents.
- databricks-jobs — async deploy pattern (
--no-wait, TASK run_id trap). - databricks-unity-catalog — UC governs the registered model: permissions, lineage, audit.
Databricks MLflow evaluation
Read databricks-core for authentication and workspace discovery. For tracing,
trace retrieval, observability, and trace-based debugging, use the separate
databricks-mlflow-tracing skill. Verify tracing when the selected evaluation mode or scorers require it; reuse
precomputed outputs or existing traces without rerunning the agent.
The guides below are references, not separately installed skills. When a guide names an upstream MLflow skill, use the matching guide here or the tracing skill.
| Request | Read this guide |
|---|---|
| Define quality criteria or scorers | Scorer design |
| Create a dataset and evaluate an agent | MLflow evaluation |
| Change agent behavior and verify improvement | Improve an agent |
For MLflow API details, consult the existing databricks-docs skill and
https://mlflow.org/docs/latest/llms.txt.
For Databricks judge models, prefer databricks:/system.ai.<model> with a
mlflow[databricks]>=3.16.0. A 404 commonly indicates an older client that
does not understand the system model URI; a 403 on a legacy endpoint name can
mean legacy serving endpoints are disabled. Verify the installed client and
available system.ai model before changing credentials or permissions.
For a new evaluation, establish a baseline with a fixed dataset and scorers,
change one thing, then run the same evaluation again. Use
databricks-mlflow-tracing first if traces are missing or a failure needs a
trace-grounded diagnosis.
Databricks MLflow Tracing
Use this skill for MLflow observability and tracing workflows on Databricks.
Read databricks-core first for CLI authentication, profile selection, and
workspace discovery.
On Databricks, reuse the existing experiment ID or choose an absolute workspace
path such as /Users/<email>/my-experiment or /Shared/my-experiment; relative
names in the bundled onboarding examples are for non-Databricks tracking servers.
Read the matching guide directly:
| Request | Read this guide |
|---|---|
| Choose a use case or get started with MLflow | MLflow onboarding |
| Instrument an app or agent and verify tracing | MLflow tracing |
| Retrieve traces by ID, time, status, or metadata | Retrieve traces |
| Inspect one trace or explain an agent failure | Analyze a trace |
| Reconstruct or debug a multi-turn chat session | Analyze a chat session |
| Query token, latency, or quality metrics on a supported OSS MLflow server | Query MLflow metrics |
For agent evaluation, scorer creation, or implementing a fix after diagnosis,
use databricks-mlflow-evaluation.
For work that spans multiple guides, use this order:
- Determine the use case with MLflow onboarding.
- Add or verify tracing with MLflow tracing.
- Retrieve traces before trace or chat-session analysis.
- Use Query MLflow metrics for aggregate trends.
For Databricks trace storage, read the tracing guide's Databricks reference before claiming that traces use Unity Catalog storage.
For a Unity Catalog trace location, verify the deployed identity rather than a
local user. It needs experiment access, CAN USE on the SQL warehouse,
USE CATALOG and USE SCHEMA, and explicit MODIFY, SELECT on each generated
spans, logs, metrics, and annotations table. ALL PRIVILEGES alone does not
satisfy those table grants. Confirm persistence by reading back a fresh trace
written under the deployed identity.
Model Serving Endpoints
FIRST: Use the parent databricks-core skill for CLI basics, authentication, and profile selection.
Gateway boundary: This skill owns legacy AI Gateway configuration attached to a Model Serving endpoint, such as
serving-endpoints put-ai-gateway. An explicit Unity Gateway / Unity AI Gateway request or anydatabricks ai-gatewaymodel service, MCP service, or model provider service belongs todatabricks-unity-gateway.
Model Serving provides managed endpoints for serving LLMs, custom ML models, and external models as scalable REST APIs. Endpoints are identified by name (unique per workspace).
Endpoint Types
| Type | When to Use | Key Detail |
|---|---|---|
| Pay-per-token | Foundation Model APIs (Llama, GPT-5, Claude, Gemini, etc.) | Uses system.ai.* catalog models, pre-provisioned in every workspace. Discover at runtime — see Foundation Model API endpoints below. |
| Provisioned throughput | Dedicated GPU capacity | Guaranteed throughput, higher cost |
| Custom model | Your own MLflow models or containers | Deploy any model with an MLflow signature |
Endpoint Structure
Serving Endpoint (top-level, identified by NAME)
├── Config
│ ├── Served Entities (model references + scaling config)
│ └── Traffic Config (routing percentages across entities)
├── AI Gateway (rate limits, usage tracking)
└── State (READY / NOT_READY, config_update status)
- Served Entities: Each entity references a model (from Unity Catalog or MLflow) with scaling parameters. Get the entity name from
served_entities[].namein thegetoutput — needed forbuild-logsandlogscommands. - Traffic Config: Routes requests across served entities by percentage (for A/B testing, canary deployments).
- State: Endpoints transition
NOT_READY→READYafter creation or config update. Poll viagetto checkstate.ready.
CLI Discovery — ALWAYS Do This First
Do NOT guess command syntax. Discover available commands and their usage dynamically:
# List all serving-endpoints subcommands
databricks serving-endpoints -h
# Get detailed usage for any subcommand (flags, args, JSON fields)
databricks serving-endpoints <subcommand> -h
Run databricks serving-endpoints -h before constructing any command. Run databricks serving-endpoints <subcommand> -h to discover exact flags, positional arguments, and JSON spec fields for that subcommand.
Create an Endpoint
Do NOT list endpoints before creating.
databricks serving-endpoints create \
--json '{
"name": "<ENDPOINT_NAME>",
"served_entities": [{
"entity_name": "<MODEL_CATALOG_PATH>",
"entity_version": "<VERSION>",
"min_provisioned_throughput": 0,
"max_provisioned_throughput": 0,
"workload_size": "Small",
"scale_to_zero_enabled": true
}],
"traffic_config": {
"routes": [{
"served_entity_name": "<ENTITY_NAME>",
"traffic_percentage": 100
}]
}
}' --profile <PROFILE>
- Discover available Foundation Models: see Foundation Model API endpoints below for the runtime-list snippet and default-picking rules. You can also check the
system.aicatalog in Unity Catalog, or rundatabricks serving-endpoints list --profile <PROFILE>to see what's deployed in the workspace. Usedatabricks serving-endpoints get-open-api <ENDPOINT_NAME> --profile <PROFILE>to inspect a specific endpoint's API schema. - Long-running operation; the CLI waits for completion by default. Use
--no-waitto return immediately, then poll:databricks serving-endpoints get <ENDPOINT_NAME> --profile <PROFILE> # Check: state.ready == "READY" - For provisioned throughput or custom model endpoints, run
databricks serving-endpoints create -hto discover the required JSON fields for your endpoint type.
MLflow Deployments client (Python alternative)
mlflow.deployments.get_deploy_client("databricks").create_endpoint(name=..., config={...}) takes the same JSON shape as the CLI. Two gotchas:
tags=is a top-level kwarg, NOT a field insideconfig. Same[{key, value}]shape asserving-endpoints patch --add-tags.traffic_config.routes[].served_model_name="<model>-<version>"(e.g."turbine_failure-3"). The API auto-derives this from the entity, but you reference the exact string intraffic_config— get the format wrong and the route silently doesn't match.
Zero-downtime version swap
To roll an endpoint to a new model version: repoint the alias and call update_endpoint with the new served_entities + matching traffic_config. Missing either half is the common bug — alias-only doesn't update the endpoint; update_endpoint-only leaves the alias pointing at the old version.
from mlflow.tracking import MlflowClient
from mlflow.deployments import get_deploy_client
registry = MlflowClient(registry_uri="databricks-uc")
deploy = get_deploy_client("databricks")
registry.set_registered_model_alias(FULL_NAME, "prod", new_version)
deploy.update_endpoint(endpoint=ENDPOINT_NAME, config={
"served_entities": [{"entity_name": FULL_NAME, "entity_version": new_version,
"workload_size": "Small", "scale_to_zero_enabled": True}],
"traffic_config": {"routes": [
{"served_model_name": f"{NAME}-{new_version}", "traffic_percentage": 100}
]},
})
The CLI equivalent is databricks serving-endpoints update-config <NAME> --json '...'. Either way, poll both state.ready and state.config_update afterward — see Endpoint Readiness below.
Endpoint Readiness
After create or update-config, the endpoint provisions compute and loads the model. Do not query the endpoint until it is ready. Two state fields matter and they mean different things:
state.ready—READYonce the endpoint has any working config. StaysREADYduring a version swap.state.config_update—NOT_UPDATINGonce the current config update finishes;IN_PROGRESSduring a version swap.
A loop watching only state.ready will say "ready" mid version-swap while the old version is still serving. Poll both:
databricks serving-endpoints get <ENDPOINT_NAME> --profile <PROFILE> \
| jq '{ready: .state.ready, config_update: .state.config_update}'
# Fully ready when ready == "READY" AND config_update == "NOT_UPDATING"
Provisioning may take several minutes. Provisioned throughput endpoints take the longest (GPU allocation). Queries to endpoints that are not yet READY return 404 or 503.
Query an Endpoint
Chat / agent endpoints use the messages array:
databricks serving-endpoints query <ENDPOINT_NAME> \
--json '{"messages": [{"role": "user", "content": "Hello"}]}' --profile <PROFILE>
Classical-ML endpoints use dataframe_records (one record per row):
databricks serving-endpoints query <ENDPOINT_NAME> \
--json '{"dataframe_records": [{"vibration": 0.42, "rpm": 18.3, "temp_c": 71.2}]}'
- Use
--streamfor streaming responses on chat endpoints. - For embeddings or other custom schemas: use
get-open-api <ENDPOINT_NAME>first to discover the request/response shape.
Get Endpoint Schema (OpenAPI)
Returns the OpenAPI 3.1 JSON schema describing what each served model accepts and returns. Use this to understand an endpoint's input/output format before querying it.
databricks serving-endpoints get-open-api <ENDPOINT_NAME> --profile <PROFILE>
The schema shows paths per served model (e.g., /served-models/<model-name>/invocations) with full request/response definitions including parameter types, enums, and nullable fields.
Other Commands
Run databricks serving-endpoints <subcommand> -h for usage details.
| Task | Command | Notes |
|---|---|---|
| List all endpoints | list | |
| Get endpoint details | get <NAME> | Shows state, config, served entities |
| Delete endpoint | delete <NAME> | |
| Update served entities or traffic | update-config <NAME> --json '...' | Zero-downtime: old config serves until new is ready |
| Rate limits & usage tracking | put-ai-gateway <NAME> --json '...' | |
| Update tags | patch <NAME> --json '...' | |
| Build logs | build-logs <NAME> <SERVED_MODEL> | Get SERVED_MODEL from get output: served_entities[].name |
| Runtime logs | logs <NAME> <SERVED_MODEL> | |
| Metrics (Prometheus format) | export-metrics <NAME> | |
| Permissions | get-permissions <ENDPOINT_ID> | ⚠️ Uses endpoint ID (hex string), not name. Find ID via get. |
What's Next
Integrate with a Databricks App
After creating a serving endpoint, wire it into a Databricks App.
Step 1 — Check if the serving plugin is available in the AppKit template:
databricks apps manifest --profile <PROFILE>
If the output includes a serving plugin, scaffold with:
databricks apps init --name <APP_NAME> \
--features serving \
--set "serving.serving-endpoint.name=<ENDPOINT_NAME>" \
--run none --profile <PROFILE>
Step 2 — If no serving plugin, add the endpoint resource manually to an existing app's databricks.yml:
resources:
apps:
my_app:
resources:
- name: my-model-endpoint
serving_endpoint:
name: <ENDPOINT_NAME>
permission: CAN_QUERY
And inject the endpoint name as an environment variable in app.yaml:
env:
- name: SERVING_ENDPOINT
valueFrom: serving-endpoint
Then wire the endpoint into your app via the serving() plugin or a custom route in onPluginsReady. For the full app integration pattern, use the databricks-apps skill and read the Model Serving Guide.
Develop & deploy new models
This skill is ops-focused (manage existing endpoints). For the dev-side flow — training, MLflow tracking, UC registration, custom PyFunc authoring, and hand-rolled ResponsesAgent code — see databricks-ml-training (experimental).
Foundation Model API endpoints
Pay-per-token, pre-provisioned in every workspace. New models land regularly and a static skill list goes stale fast — always list at runtime instead of hard-coding names. Filter by the databricks- name prefix AND by the served entity being in system.ai.* (other endpoints like databricks-app-template-serving share the prefix but aren't FM API endpoints). Use Databricks Unity Gateway (databricks-unity-gateway) to manage access to foundation model endpoints. Do not use databricks-model-serving legacy AI Gateway.
# FM API endpoints in this workspace, grouped by task (chat / embeddings / etc.)
databricks serving-endpoints list \
| jq -r '.[]
| select(.name | startswith("databricks-"))
| select((.config.served_entities[0].entity_name // "") | startswith("system.ai."))
| "\(.task)\t\(.name)"' \
| sort
Defaults when the user doesn't specify: pick the highest-numbered Claude Sonnet for agents, the highest-numbered -codex-max for code, databricks-gte-large-en for embeddings — resolve actual names from the live list above.
Off-platform streaming
For apps deployed outside Databricks Apps (Vercel, AWS, standalone Node.js) hitting Databricks legacy AI Gateway with Vercel AI SDK v6, see references/off-platform-streaming.md. Off-platform apps should be updated to hit Databricks Unity Gateway (databricks-unity-gateway). For AppKit-based apps, use the databricks-apps skill's built-in serving plugin instead.
Troubleshooting
| Error | Solution |
|---|---|
cannot configure default credentials | Use --profile flag or authenticate first |
PERMISSION_DENIED | Check workspace permissions; for apps, ensure serving_endpoint resource declared with CAN_QUERY |
Endpoint stuck in NOT_READY | Wait up to 30 min for provisioned throughput. Check build logs: build-logs <NAME> <ENTITY_NAME> (get entity name from get output → served_entities[].name) |
RESOURCE_DOES_NOT_EXIST | Verify endpoint name with list |
| Query returns 404 | Endpoint may still be provisioning; check state.ready via get |
RATE_LIMIT_EXCEEDED (429) | AI Gateway rate limit; check put-ai-gateway config or retry after backoff |
| Endpoint missing from the Serving UI after deploy | UI filter defaults to "Owned by me". Deploy jobs run as a service principal, so the endpoint is hidden until you switch to "All". databricks serving-endpoints list always shows it. |
Lakeflow Spark Declarative Pipelines Development
FIRST: Use the parent databricks-core skill for CLI basics, authentication, profile selection, and data discovery commands.
SDP, LDP, Lakeflow Declarative Pipelines, and DLT (former name) all refer to the same product. This skill covers the core syntax and rules for writing Spark Declarative Pipelines datasets. For file ingestion and document parsing read file-ingestion; for failed, slow, or flaky updates read pipeline-debugging; for migrating dbt or Informatica projects read migrate-to-sdp.
Decision Tree
Use this tree to choose the dataset type first, then layer on applicable features. Multiple features can apply to the same dataset — e.g., a Streaming Table can use Auto Loader for ingestion, Append Flows for fan-in, and Expectations for data quality.
User request → What kind of output?
├── Intermediate/reusable logic (not persisted) → Temporary View (default)
│ ├── Preprocessing/filtering before Auto CDC → Temporary View feeding CDC flow
│ ├── Shared intermediate streaming logic reused by multiple downstream tables
│ ├── Pipeline-private helper logic (not published to catalog)
│ ├── Expensive intermediate that is reused across refreshes → Private ST/MV (materialized but pipeline-scoped)
│ └── Published to UC for external queries → Persistent View (SQL only)
├── Persisted dataset
│ ├── Source is a periodic complete snapshot that must be diffed by key → Streaming Table + Auto CDC FROM SNAPSHOT
│ ├── Source is streaming/incremental/continuously growing → Streaming Table
│ │ ├── File ingestion (cloud storage, Volumes) → Auto Loader (see [file-ingestion](references/file-ingestion.md))
│ │ ├── Message bus (Kafka, Kinesis, Pub/Sub, Pulsar, Event Hubs) → streaming source read
│ │ ├── Existing streaming/Delta table → streaming read from table
│ │ ├── CDC / upserts / track changes / keep latest per key / SCD Type 1, 2, or bitemporal → Auto CDC (only when this pipeline owns the target; merge/upsert into an existing/external table → ForEachBatch Sink, see Sink branch)
│ │ ├── Source rows carry two timestamps with different semantic meaning — one for when the fact holds in the domain, the other for when the system learned or recorded it (e.g. effective_at + ingested_at, trade date + booking date, observation time + chart-entry time) → Auto CDC STORED AS BITEMPORAL (Beta, DBR 17.3+); read the Auto CDC reference before recommending.
│ │ ├── Multiple sources into one table → Append Flows (NOT union)
│ │ ├── Historical backfill + live stream → one-time Append Flow + regular flow
│ │ ├── Recompute/overwrite a bounded window or partition each run (late-arriving within a known lookback, rolling last-N-days aggregate, partition overwrite, scheduled re-runs) → REPLACE WHERE flow (NOT a Materialized View, plain streaming table, or full recompute)
│ │ ├── Streaming source re-emits all rows for a key and the latest should replace the old per key (re-ingesting mutable files, keyed in-place updates) → REPLACE USING flow (NOT Auto CDC, which is for change events/SCD; NOT REPLACE WHERE, which overwrites a predicate window not a key)
│ │ ├── Windowed aggregation with watermark → stateful streaming
│ │ └── Custom per-key state, timers, timeouts (Python) → transformWithState in a flow (timeMode EventTime or None in triggered pipelines)
│ └── Source is batch/historical/full scan → Materialized View
│ ├── Aggregation/join across full dataset (GROUP BY, SUM, COUNT, etc.)
│ ├── Gold layer aggregation from streaming table → MV with batch read (spark.read / no STREAM)
│ ├── JDBC/Federation/external batch sources
│ └── Small static file load (reference data, no streaming read)
├── Output to a target the pipeline does NOT own (external / pre-existing / managed elsewhere) → Sink (Python only, primary even if the user asked for SQL)
│ ├── External or pre-existing Delta table NOT owned by this pipeline → Sink with format="delta" (streaming tables/MVs are for tables this pipeline owns and manages — including ones it already created and refreshes each run — but they cannot write into a table owned outside the pipeline)
│ ├── Kafka / Event Hubs → Sink with format="kafka" + @dp.append_flow(target="sink_name")
│ ├── Custom destination not natively supported → Sink with custom format
│ ├── Custom merge/upsert into an existing/external table → ForEachBatch Sink, NOT Auto CDC
│ └── Multiple destinations per batch → ForEachBatch Sink
└── Data quality constraints → Expectations (on any dataset type)
Common Traps
- Changing dataset type (ST↔MV): Read changing-dataset-type BEFORE edits or commands. Create a NEW dataset name and repoint ALL downstream references; never convert in place. Full refresh cannot fix this. The platform's "drop first" message is a manual workaround; offer dropping only as a follow-up. A green dry run does not prove rewiring.
- SCD Type 2 queries: Use validity columns
__START_AT/__END_AT, not invented names. Current rows:__END_AT IS NULL. Point-in-time predicate:__START_AT <= t AND (__END_AT > t OR __END_AT IS NULL); temporal joins also match the key and use the event date ast. Read the Auto CDC reference (Python, SQL) and scd-2-querying. - UNRESOLVED_COLUMN / UNRESOLVED_COLUMN.WITH_SUGGESTION: Verify upstream columns. For Auto Loader, verify its schema includes them (e.g., via
cloudFiles.schemaHints/schemaHints); read auto-loader-schema. - Existing ST column add/drop/rename/widen, DELTA_MERGE_INCOMPATIBLE_DATATYPE or DELTA_FAILED_TO_MERGE_FIELDS: Read streaming-table-schema-evolution before answering;
ALTER TABLEis Beta with prerequisites. - "Create a table" without a type: Ask whether the source is streaming or batch. Streaming → Streaming Table; batch → Materialized View. Mismatched pairs fail validation.
- Aggregating a streaming source: For per-window results (hourly, daily) from an append-only source, use a streaming table with a
WATERMARKandwindow(), which bounds state (SQL / Python streaming patterns). Otherwise use a Materialized View with a batch read (noSTREAM): streaming tables are append-only and don't recompute aggregates when source rows change. Cannot create streaming table from batch query: The streaming table reads in batch. UseFROM STREAM read_files(...)/FROM STREAM(table).- Python-only features (sinks, ForEachBatch sinks, custom data sources, Real-Time Mode): When the user works in SQL, say so and suggest Python for that part.
databricks fson Volumes: The CLI needs thedbfs:prefix, e.g.databricks fs ls dbfs:/Volumes/<catalog>/<schema>/<volume>/<path>.- SQL parameters (DBR 17.0+): Read parameters-sql before writing. Use
:namefor values that vary across updates, not hardcoding or legacy${name}. Values areSTRING: cast at use; identifiers needIDENTIFIER(). - Recommend one approach. Omit anti-patterns and inferior alternatives; mention alternatives only for viable trade-offs.
Publishing Modes
Pipelines publish to the default catalog/schema in their settings.
- Use dataset-only names for pipeline reads/definitions unless existing code or the user requests qualification.
catalog.schema.tablepublishes a pipeline-owned dataset elsewhere; no sink needed. External, pre-existing, or separately managed targets require a Python sink, even if the user requested SQL.USE CATALOG/USE SCHEMAchanges the defaults for subsequent definitions in the same SQL file.
Comprehensive API Reference
MANDATORY: Before implementing, editing, or suggesting feature code, read its linked reference and these required companion references:
- Auto Loader, Auto CDC, sinks → also read the streaming-table reference: Auto Loader/CDC target streaming tables; sinks use streaming append flows.
- Auto Loader → also read the matching format options: CSV, JSON, Avro, Parquet, XML, Text, ORC. For schema inference/evolution, hints, rescued data, partition inference or type widening, read schema.
- Expectations → also read the target dataset's definition reference for constraint placement.
- Kafka / Event Hubs sources → also read authentication: Python or SQL (SASL, MSK IAM, Entra ID, GCP OAuth; shaded class names required).
- Avro with Confluent Schema Registry → read Kafka deserialization, Python only. For SQL pipelines, explain the limitation and recommend switching to Python. Protobuf also works in SQL (
from_protobuf, DBR 16.1+; see the SQL deserialization reference). - JSON/Avro without Schema Registry → Python or SQL, matching the pipeline language.
- Avro with Confluent Schema Registry → read Kafka deserialization, Python only. For SQL pipelines, explain the limitation and recommend switching to Python. Protobuf also works in SQL (
- Real-Time Mode → read reference. Only for explicitly requested operational sub-second latency; requires continuous mode + a real-time update flow.
Feature references
| Feature | Python | SQL |
|---|---|---|
| Streaming tables; explicit targets; append and one-time backfill flows; skip upstream change commits | Python | SQL |
| Materialized views; manual/auto liquid clustering, partitioning, table properties, explicit schemas, generated columns, row filters and column masks, private datasets | Python | SQL |
| Temporary views | Python | SQL |
| Persistent UC views | Unsupported | SQL |
| Delta/Kafka/Event Hubs/custom sinks | Python | Unsupported |
| ForEachBatch sinks | Python | Unsupported |
| Auto CDC from a stream | Python | SQL |
| Auto CDC from periodic snapshots | Python | SQL |
| Expectations: warn, drop, fail; individual or multiple constraints | Python | SQL |
| Identity columns | Python | SQL |
| Auto Loader | Python | SQL |
| Kafka/Event Hubs sources | Python | SQL |
| Kinesis sources | Python | SQL |
| Payload deserialization (JSON/Avro/Protobuf; Avro with Schema Registry is Python only) | Python | SQL |
- REPLACE WHERE: overwrite a bounded window/partition each run, such as a rolling lookback. Read Python or SQL.
- REPLACE USING: streaming keyed replacement of full rows with a required sequencing column; Beta, requires DBR 18.2+. Read Python or SQL.
Topics covering both languages:
- Streaming patterns (SQL / Python): dedup, windowed aggregations, late data, stream-stream joins, rescued-data quarantine, lag monitoring,
transformWithState. - SCD Type 2 querying: current state, point-in-time, joining facts with historical dimensions.
Reads and compatibility
- Pipeline batch reads:
spark.read.table("name")/SELECT ... FROM name; streaming reads:spark.readStream.table("name")/SELECT ... FROM STREAM name. - Pub/Sub and Pulsar sources: use
spark.readStream.format("pubsub")orspark.readStream.format("pulsar"); SQL usesSTREAM read_pubsub(...)orSTREAM read_pulsar(...). - JDBC/Federation are batch sources; Python uses
spark.read.format("postgresql")etc. and SQL references the federation catalog. - Static files:
spark.read.format(...).load(...)/read_files(...)withoutSTREAM. - Custom data sources are Python-only:
spark.read[Stream].format("custom"). - Prefer
from pyspark import pipelines as dpfor new Python. Preserve existingdltimports. Preferspark.read.table/spark.readStream.tableover the legacydp.read,dlt.read, and their streaming variants. - Never use
LIVE.references or legacyCREATE LIVE TABLE/CREATE LIVE VIEWsyntax; use current dataset definitions. The one exception isCREATE LIVE VIEWfor expectations on a temporary view (see the table below).
Migrating legacy DLT syntax
When the user asks to migrate legacy DLT code to the modern form, read dlt-migration.md before suggesting changes so the conversion is correct (especially around apply_changes → create_auto_cdc_flow semantics and partition_cols → cluster_by). Common mappings:
| If you see… | …migrate to |
|---|---|
import dlt | from pyspark import pipelines as dp |
@dlt.table(...), @dlt.append_flow(...), @dlt.expect* | Same decorator name on dp.* (e.g. @dp.table, @dp.expect_or_drop). |
@dlt.view(...) (or @dp.view(...) if present in older code) | @dp.temporary_view(...) |
dlt.read("name") / dlt.read_stream("name") | spark.read.table("name") / spark.readStream.table("name") |
dp.read(...) / dp.read_stream(...) | Also legacy — use spark.read.table(...) / spark.readStream.table(...). |
dlt.apply_changes(...) / dp.apply_changes(...) | dp.create_auto_cdc_flow(...). sequence_by accepts a column name (string) or col(...); stored_as_scd_type is 1 or 2 (integer or string). |
dlt.apply_changes_from_snapshot(...) | dp.create_auto_cdc_from_snapshot_flow(...) |
dlt.create_streaming_table(...) | dp.create_streaming_table(...) |
LIVE.<name> prefix in SQL | Bare name (SELECT FROM name for batch, SELECT FROM STREAM(name) for streaming). LIVE. will error in modern pipelines. |
CREATE LIVE TABLE / CREATE STREAMING LIVE TABLE | CREATE OR REFRESH MATERIALIZED VIEW / CREATE OR REFRESH STREAMING TABLE. |
CREATE TEMPORARY LIVE VIEW (a.k.a. CREATE LIVE VIEW) | CREATE TEMPORARY VIEW. Exception: CREATE TEMPORARY VIEW does NOT support CONSTRAINT clauses for expectations — for the rare case where you need expectations on a temp view, CREATE LIVE VIEW is retained. See temporary-view-sql.md and expectations-sql.md. |
APPLY CHANGES INTO ... FROM STREAM ... (SQL) | AUTO CDC INTO ... FROM STREAM ... |
partition_cols=[...] / PARTITIONED BY (...) + ZORDER | cluster_by=[...] / CLUSTER BY (...) (Liquid Clustering). |
input_file_name() | _metadata.file_path (SQL) / F.col("_metadata.file_path") (Python). |
target=... parameter on create_streaming_table / pipeline config | schema=... |
Performance
- Performance tuning - Liquid clustering, cluster key selection by layer, streaming state management, join optimization, query optimization, and pre-aggregation
- Making a materialized view with more than 3 joins incremental → decompose-materialized-view-python (Python) or decompose-materialized-view-sql (SQL)
Language-specific guides
- Before writing Python, first read Python basics: decorators, functions, and critical rules.
- Before writing SQL, first read SQL basics: statements and critical rules.
Project Setup
- Language: Keep an existing pipeline's language. For new pipelines default to SQL; use Python when the user asks for it or needs UDFs, ML inference, or Python-only features. If the request is ambiguous, ask.
- New pipeline: Read setup-pipeline for how to scope and build it.
- Workflow: Pick before scaffolding.
- A: Standalone project (DAB), no existing
databricks.yml→ 1-project-initialization-with-dab. Default for production-bound work. - B: Pipeline in an existing bundle → add
resources/<name>.pipeline.yml; see Workflow B in the same reference. - C: Rapid CLI iteration (no bundle) for prototypes and demos → 2-rapid-iteration-with-cli.
- A: Standalone project (DAB), no existing
- Structure: Medallion (Bronze → Silver → Gold) with just a few tables unless the user says otherwise. One dataset per file, named after the dataset, in
src/ortransformations/. In Gold, keep the dimensions consumers filter by (e.g., every dashboard filter). - Settings: For create/update JSON, compute, channels, and multi-schema setups, read pipeline-settings.
Running a Pipeline
- DAB (A/B): Code changes take effect only after
databricks bundle deploy; deploy before any run, dry run, or selective refresh. Sequence:databricks bundle validate,databricks bundle deploy -t dev,databricks bundle run <pipeline_name> -t dev. - CLI (C): Re-upload with
databricks workspace import-dir ... --overwriteafter every change, thendatabricks pipelines start-update. - Poll the update ID, not the top-level pipeline state, as described in the workflow reference. A serverless update can stay
INITIALIZINGfor a few minutes on cold start; don't cancel it. - Dry run (
--validate-only) is a faster way to confirm the correctness of the pipeline code. - Selective refresh is preferred when you only need to run one table; its dependencies must already be materialized.
- Full refresh is the most expensive and dangerous option and can lead to data loss (it resets streaming state); use it only when really necessary, and always as a follow-up the user must explicitly select.
- Terminology: Never say "run a dry run" — the phrase is redundant. Say "do a dry run" or "start a dry run" instead.
- To perform a rewind pipeline operation, first read rewind-pipeline.
- To refresh or reset-and-refresh a single flow of a table (rather than the whole table), first read flow-reset-refresh. Resetting a flow cannot be undone; like a full refresh, the user must explicitly select it.
Pipeline Settings
- To answer questions about or change pipeline settings, read pipeline-settings.md.
- Make a pipeline continuous / always-on / "run continuously", or set its maintenance window → do NOT just set
continuous: true; read continuous-pipeline.md first, as the mechanism depends on the pipeline's current mode.
Databricks Development Guide
This skill provides guidance for Databricks SDK, Databricks Connect, CLI, and REST API.
SDK Documentation: https://databricks-sdk-py.readthedocs.io/en/latest/ GitHub Repository: https://github.com/databricks/databricks-sdk-py
Environment Setup
- Use existing virtual environment at
.venvor useuvto create one - For Spark operations:
uv pip install databricks-connect - For SDK operations:
uv pip install databricks-sdk - Databricks CLI version should be 0.278.0 or higher
Configuration
- Default profile name:
DEFAULT - Config file:
~/.databrickscfg - Environment variables:
DATABRICKS_HOST,DATABRICKS_TOKEN
Databricks Connect (Spark Operations)
Use databricks-connect for running Spark code locally against a Databricks cluster.
from databricks.connect import DatabricksSession
# Auto-detects 'DEFAULT' profile from ~/.databrickscfg
spark = DatabricksSession.builder.getOrCreate()
# With explicit profile
spark = DatabricksSession.builder.profile("MY_PROFILE").getOrCreate()
# Use spark as normal
df = spark.sql("SELECT * FROM catalog.schema.table")
df.show()
IMPORTANT: Do NOT set .master("local[*]") - this will cause issues with Databricks Connect.
Direct REST API Access
For operations not yet in SDK or overly complex via SDK, use direct REST API:
from databricks.sdk import WorkspaceClient
w = WorkspaceClient()
# Direct API call using authenticated client
response = w.api_client.do(
method="GET",
path="/api/2.0/clusters/list"
)
# POST with body
response = w.api_client.do(
method="POST",
path="/api/2.0/jobs/run-now",
body={"job_id": 123}
)
When to use: Prefer SDK methods when available. Use api_client.do for:
- New API endpoints not yet in SDK
- Complex operations where SDK abstraction is problematic
- Debugging/testing raw API responses
Databricks CLI
# Check version (should be >= 0.278.0)
databricks --version
# Use specific profile
databricks --profile MY_PROFILE clusters list
# Common commands
databricks clusters list
databricks jobs list
databricks workspace list /Users/me
SDK Reference
For a per-API table of method signatures and direct doc URLs (Clusters, Jobs, SQL warehouses, Unity Catalog, Serving, Vector Search, etc.), see references/doc-index.md. Worked examples for the main API surfaces live under examples/.
SDK Documentation Architecture
The SDK documentation follows a predictable URL pattern:
Base: https://databricks-sdk-py.readthedocs.io/en/latest/
Workspace APIs: /workspace/{category}/{service}.html
Account APIs: /account/{category}/{service}.html
Authentication: /authentication.html
DBUtils: /dbutils.html
Workspace API Categories
| Category | Services |
|---|---|
compute | clusters, cluster_policies, command_execution, instance_pools, libraries |
catalog | catalogs, schemas, tables, volumes, functions, storage_credentials, external_locations |
jobs | jobs |
sql | warehouses, statement_execution, queries, alerts, dashboards |
serving | serving_endpoints |
vectorsearch | vector_search_indexes, vector_search_endpoints |
pipelines | pipelines |
workspace | repos, secrets, workspace, git_credentials |
files | files, dbfs |
ml | experiments, model_registry |
Authentication
Doc: https://databricks-sdk-py.readthedocs.io/en/latest/authentication.html
Environment Variables
DATABRICKS_HOST=https://your-workspace.cloud.databricks.com
DATABRICKS_TOKEN=dapi... # Personal Access Token
Code Patterns
# Auto-detect credentials from environment
from databricks.sdk import WorkspaceClient
w = WorkspaceClient()
# Explicit token auth
w = WorkspaceClient(
host="https://your-workspace.cloud.databricks.com",
token="dapi..."
)
# Azure Service Principal
w = WorkspaceClient(
host="https://adb-xxx.azuredatabricks.net",
azure_workspace_resource_id="/subscriptions/.../resourceGroups/.../providers/Microsoft.Databricks/workspaces/...",
azure_tenant_id="tenant-id",
azure_client_id="client-id",
azure_client_secret="secret"
)
# Use a named profile from ~/.databrickscfg
w = WorkspaceClient(profile="MY_PROFILE")
Core API Reference
Clusters API
Doc: https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html
# List all clusters
for cluster in w.clusters.list():
print(f"{cluster.cluster_name}: {cluster.state}")
# Get cluster details
cluster = w.clusters.get(cluster_id="0123-456789-abcdef")
# Create a cluster (returns Wait object)
wait = w.clusters.create(
cluster_name="my-cluster",
spark_version=w.clusters.select_spark_version(latest=True),
node_type_id=w.clusters.select_node_type(local_disk=True),
num_workers=2
)
cluster = wait.result() # Wait for cluster to be running
# Or use create_and_wait for blocking call
cluster = w.clusters.create_and_wait(
cluster_name="my-cluster",
spark_version="14.3.x-scala2.12",
node_type_id="i3.xlarge",
num_workers=2,
timeout=timedelta(minutes=30)
)
# Start/stop/delete
w.clusters.start(cluster_id="...").result()
w.clusters.stop(cluster_id="...")
w.clusters.delete(cluster_id="...")
Jobs API
Doc: https://databricks-sdk-py.readthedocs.io/en/latest/workspace/jobs/jobs.html
from databricks.sdk.service.jobs import Task, NotebookTask
# List jobs
for job in w.jobs.list():
print(f"{job.job_id}: {job.settings.name}")
# Create a job
created = w.jobs.create(
name="my-job",
tasks=[
Task(
task_key="main",
notebook_task=NotebookTask(notebook_path="/Users/me/notebook"),
existing_cluster_id="0123-456789-abcdef"
)
]
)
# Run a job now
run = w.jobs.run_now_and_wait(job_id=created.job_id)
print(f"Run completed: {run.state.result_state}")
# Get run output
output = w.jobs.get_run_output(run_id=run.run_id)
SQL Statement Execution
Doc: https://databricks-sdk-py.readthedocs.io/en/latest/workspace/sql/statement_execution.html
# Execute SQL query
response = w.statement_execution.execute_statement(
warehouse_id="abc123",
statement="SELECT * FROM catalog.schema.table LIMIT 10",
wait_timeout="30s"
)
# Check status and get results
if response.status.state == StatementState.SUCCEEDED:
for row in response.result.data_array:
print(row)
# For large results, fetch chunks
chunk = w.statement_execution.get_statement_result_chunk_n(
statement_id=response.statement_id,
chunk_index=0
)
SQL Warehouses
Doc: https://databricks-sdk-py.readthedocs.io/en/latest/workspace/sql/warehouses.html
# List warehouses
for wh in w.warehouses.list():
print(f"{wh.name}: {wh.state}")
# Get warehouse
warehouse = w.warehouses.get(id="abc123")
# Create warehouse
created = w.warehouses.create_and_wait(
name="my-warehouse",
cluster_size="Small",
max_num_clusters=1,
auto_stop_mins=15
)
# Start/stop
w.warehouses.start(id="abc123").result()
w.warehouses.stop(id="abc123").result()
Unity Catalog - Tables
Doc: https://databricks-sdk-py.readthedocs.io/en/latest/workspace/catalog/tables.html
# List tables in a schema
for table in w.tables.list(catalog_name="main", schema_name="default"):
print(f"{table.full_name}: {table.table_type}")
# Get table info
table = w.tables.get(full_name="main.default.my_table")
print(f"Columns: {[c.name for c in table.columns]}")
# Check if table exists
exists = w.tables.exists(full_name="main.default.my_table")
Unity Catalog - Catalogs & Schemas
Doc (Catalogs): https://databricks-sdk-py.readthedocs.io/en/latest/workspace/catalog/catalogs.html Doc (Schemas): https://databricks-sdk-py.readthedocs.io/en/latest/workspace/catalog/schemas.html
# List catalogs
for catalog in w.catalogs.list():
print(catalog.name)
# Create catalog
w.catalogs.create(name="my_catalog", comment="Description")
# List schemas
for schema in w.schemas.list(catalog_name="main"):
print(schema.name)
# Create schema
w.schemas.create(name="my_schema", catalog_name="main")
Volumes
Doc: https://databricks-sdk-py.readthedocs.io/en/latest/workspace/catalog/volumes.html
from databricks.sdk.service.catalog import VolumeType
# List volumes
for vol in w.volumes.list(catalog_name="main", schema_name="default"):
print(f"{vol.full_name}: {vol.volume_type}")
# Create managed volume
w.volumes.create(
catalog_name="main",
schema_name="default",
name="my_volume",
volume_type=VolumeType.MANAGED
)
# Read volume info
vol = w.volumes.read(name="main.default.my_volume")
Files API
Doc: https://databricks-sdk-py.readthedocs.io/en/latest/workspace/files/files.html
# Upload file to volume
w.files.upload(
file_path="/Volumes/main/default/my_volume/data.csv",
contents=open("local_file.csv", "rb")
)
# Download file
with w.files.download(file_path="/Volumes/main/default/my_volume/data.csv") as f:
content = f.read()
# List directory contents
for entry in w.files.list_directory_contents("/Volumes/main/default/my_volume/"):
print(f"{entry.name}: {entry.is_directory}")
# Upload/download with progress (parallel)
w.files.upload_from(
file_path="/Volumes/main/default/my_volume/large.parquet",
source_path="/local/path/large.parquet",
use_parallel=True
)
w.files.download_to(
file_path="/Volumes/main/default/my_volume/large.parquet",
destination="/local/output/",
use_parallel=True
)
Serving Endpoints (Model Serving)
Doc: https://databricks-sdk-py.readthedocs.io/en/latest/workspace/serving/serving_endpoints.html
# List endpoints
for ep in w.serving_endpoints.list():
print(f"{ep.name}: {ep.state}")
# Get endpoint
endpoint = w.serving_endpoints.get(name="my-endpoint")
# Query endpoint
response = w.serving_endpoints.query(
name="my-endpoint",
inputs={"prompt": "Hello, world!"}
)
# For chat/completions endpoints
response = w.serving_endpoints.query(
name="my-chat-endpoint",
messages=[{"role": "user", "content": "Hello!"}]
)
# Get OpenAI-compatible client
openai_client = w.serving_endpoints.get_open_ai_client()
Vector Search
Doc (Indexes): https://databricks-sdk-py.readthedocs.io/en/latest/workspace/vectorsearch/vector_search_indexes.html Doc (Endpoints): https://databricks-sdk-py.readthedocs.io/en/latest/workspace/vectorsearch/vector_search_endpoints.html
query_index uses the workspace query URL, capped at a few tens of QPS. For apps and production traffic, use the databricks-vector-search skill (AISearchClient with a service principal and the index URL).
# List vector search indexes
for idx in w.vector_search_indexes.list_indexes(endpoint_name="my-vs-endpoint"):
print(idx.name)
# Query index
results = w.vector_search_indexes.query_index(
index_name="main.default.my_index",
columns=["id", "text", "embedding"],
query_text="search query",
num_results=10
)
for doc in results.result.data_array:
print(doc)
Pipelines (Delta Live Tables)
Doc: https://databricks-sdk-py.readthedocs.io/en/latest/workspace/pipelines/pipelines.html
# List pipelines
for pipeline in w.pipelines.list_pipelines():
print(f"{pipeline.name}: {pipeline.state}")
# Get pipeline
pipeline = w.pipelines.get(pipeline_id="abc123")
# Start pipeline update
w.pipelines.start_update(pipeline_id="abc123")
# Stop pipeline
w.pipelines.stop_and_wait(pipeline_id="abc123")
Secrets
Doc: https://databricks-sdk-py.readthedocs.io/en/latest/workspace/workspace/secrets.html
# List secret scopes
for scope in w.secrets.list_scopes():
print(scope.name)
# Create scope
w.secrets.create_scope(scope="my-scope")
# Put secret
w.secrets.put_secret(scope="my-scope", key="api-key", string_value="secret123")
# Get secret (returns GetSecretResponse with value)
secret = w.secrets.get_secret(scope="my-scope", key="api-key")
# List secrets in scope (metadata only, not values)
for s in w.secrets.list_secrets(scope="my-scope"):
print(s.key)
DBUtils
Doc: https://databricks-sdk-py.readthedocs.io/en/latest/dbutils.html
# Access dbutils through WorkspaceClient
dbutils = w.dbutils
# File system operations
files = dbutils.fs.ls("/")
dbutils.fs.cp("dbfs:/source", "dbfs:/dest")
dbutils.fs.rm("dbfs:/path", recurse=True)
# Secrets (same as w.secrets but dbutils interface)
value = dbutils.secrets.get(scope="my-scope", key="my-key")
Common Patterns
CRITICAL: Async Applications (FastAPI, etc.)
The Databricks SDK is fully synchronous. All calls block the thread. In async applications (FastAPI, asyncio), you MUST wrap SDK calls with asyncio.to_thread() to avoid blocking the event loop.
import asyncio
from databricks.sdk import WorkspaceClient
w = WorkspaceClient()
# WRONG - blocks the event loop
async def get_clusters_bad():
return list(w.clusters.list()) # BLOCKS!
# CORRECT - runs in thread pool
async def get_clusters_good():
return await asyncio.to_thread(lambda: list(w.clusters.list()))
# CORRECT - for simple calls
async def get_cluster(cluster_id: str):
return await asyncio.to_thread(w.clusters.get, cluster_id)
# CORRECT - FastAPI endpoint
from fastapi import FastAPI
app = FastAPI()
@app.get("/clusters")
async def list_clusters():
clusters = await asyncio.to_thread(lambda: list(w.clusters.list()))
return [{"id": c.cluster_id, "name": c.cluster_name} for c in clusters]
@app.post("/query")
async def run_query(sql: str, warehouse_id: str):
# Wrap the blocking SDK call
response = await asyncio.to_thread(
w.statement_execution.execute_statement,
statement=sql,
warehouse_id=warehouse_id,
wait_timeout="30s"
)
return response.result.data_array
Note: WorkspaceClient().config.host is NOT a network call - it just reads config. No need to wrap property access.
Wait for Long-Running Operations
from datetime import timedelta
# Pattern 1: Use *_and_wait methods
cluster = w.clusters.create_and_wait(
cluster_name="test",
spark_version="14.3.x-scala2.12",
node_type_id="i3.xlarge",
num_workers=2,
timeout=timedelta(minutes=30)
)
# Pattern 2: Use Wait object
wait = w.clusters.create(...)
cluster = wait.result() # Blocks until ready
# Pattern 3: Manual polling with callback
def progress(cluster):
print(f"State: {cluster.state}")
cluster = w.clusters.wait_get_cluster_running(
cluster_id="...",
timeout=timedelta(minutes=30),
callback=progress
)
Pagination
# All list methods return iterators that handle pagination automatically
for job in w.jobs.list(): # Fetches all pages
print(job.settings.name)
# For manual control
from databricks.sdk.service.jobs import ListJobsRequest
response = w.jobs.list(limit=10)
for job in response:
print(job)
Error Handling
from databricks.sdk.errors import NotFound, PermissionDenied, ResourceAlreadyExists
try:
cluster = w.clusters.get(cluster_id="invalid-id")
except NotFound:
print("Cluster not found")
except PermissionDenied:
print("Access denied")
When Uncertain
If I'm unsure about a method, I should:
-
Check the documentation URL pattern:
https://databricks-sdk-py.readthedocs.io/en/latest/workspace/{category}/{service}.html
-
Common categories:
- Clusters:
/workspace/compute/clusters.html - Jobs:
/workspace/jobs/jobs.html - Tables:
/workspace/catalog/tables.html - Warehouses:
/workspace/sql/warehouses.html - Serving:
/workspace/serving/serving_endpoints.html
- Clusters:
-
Fetch and verify before providing guidance on parameters or return types.
Quick Reference Links
Related Skills
- databricks-core - profile and authentication setup
- databricks-dabs - deploying resources via DABs
- databricks-jobs - job orchestration patterns
- databricks-unity-catalog - catalog governance
- databricks-model-serving - serving endpoint management
- databricks-vector-search - vector index operations
- databricks-lakebase - managed PostgreSQL with autoscaling + branching
Databricks Sandbox
Databricks Sandbox is a serverless, isolated compute environment for humans and agents to run untrusted code. Drive it from code with the Databricks SDK, available in Python, Java, Go, and JavaScript. Lifecycle, file operations, and synchronous execute_command_sync are available in all four SDKs; streaming execute_command is currently Python-only. The examples in this skill use the Python SDK.
SDK reference (Python): https://databricks-sdk-py.readthedocs.io/en/latest/workspace/sandbox/sandbox.html Product docs: https://docs.databricks.com/compute/serverless/sandbox SDK user guide: https://docs.databricks.com/compute/serverless/sandbox-usage-guide REST API reference: https://docs.databricks.com/api/sandbox/v1/sandbox
Other language SDKs: Java, Go, JavaScript.
For the full per-method signature table, see references/sdk-reference.md.
Environment Setup
- Install the SDK with the sandbox extra:
uv pip install "databricks-sdk[sandbox]>=0.154.0". Thesandboxextra supplies the gRPC dependencies that streamingexecute_commandneeds (the plaindatabricks-sdkpackage omits them); lifecycle, file operations, and unaryexecute_command_syncwork without the extra. 0.154.0 is the first release with the file-operation methods used below. - Authentication, profiles, and
~/.databrickscfgare covered by thedatabricks-coreskill. The sandbox client authenticates like any otherWorkspaceClient. - Your workspace must be in a supported region (see Features with limited regional availability) and have the Databricks Sandbox preview enabled on the workspace Previews page.
Resource Naming
A sandbox's resource name is sandboxes/<sandbox_id>. create_sandbox takes the bare sandbox_id; every other call takes the full name.
Lifecycle
Lifecycle calls return before the transition finishes (create_sandbox returns while PENDING, stop_sandbox while STOPPING), so poll get_sandbox(name).status.state and wait for the target state before depending on it.
import time
from databricks.sdk import WorkspaceClient
from databricks.sdk.service.sandbox import ComputeSpec, Sandbox, SandboxSpec, SandboxState
from google.protobuf.duration_pb2 import Duration
w = WorkspaceClient()
def wait_for_state(name: str, state: SandboxState, timeout: float = 300) -> Sandbox:
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
sandbox = w.sandbox.get_sandbox(name=name)
if sandbox.status and sandbox.status.state == state:
return sandbox
time.sleep(2)
raise TimeoutError(f"{name} did not reach {state}")
# Create returns while the sandbox is still PENDING; wait for RUNNING before use.
# A sandbox bills hourly while running; set inactivity_timeout so it auto-stops when
# idle.
# Sandboxes can be updated with update_sandbox. The only accepted update paths
# are display_name and spec.compute.inactivity_timeout.
created = w.sandbox.create_sandbox(
sandbox=Sandbox(spec=SandboxSpec(compute=ComputeSpec(inactivity_timeout=Duration(seconds=3600)))),
sandbox_id="my-sandbox",
)
name = created.name # "sandboxes/my-sandbox"
wait_for_state(name, SandboxState.SANDBOX_STATE_RUNNING)
# Inspect
sandbox = w.sandbox.get_sandbox(name=name)
for s in w.sandbox.list_sandboxes():
print(s.name)
# Stop keeps the home directory; it returns while STOPPING, so wait for STOPPED
# before starting again.
w.sandbox.stop_sandbox(name=name)
wait_for_state(name, SandboxState.SANDBOX_STATE_STOPPED)
w.sandbox.start_sandbox(name=name)
wait_for_state(name, SandboxState.SANDBOX_STATE_RUNNING)
# Delete is terminal.
w.sandbox.delete_sandbox(name=name)
Stopping preserves only the home directory (/home/sandbox-agent, which holds up to 100 GB); data written elsewhere does not survive a stop (up to 10 GB of non-home storage, cleared on stop). Deleting a sandbox is terminal.
A stopped sandbox (previously by stop_sandbox()) can be restarted by explicit start_sandbox() or by running execute_command / execute_command_sync against the sandbox. For sandbox creation or restart through create_sandbox() and start_sandbox(), wait for the sandbox status to reach Running before using it; execute_command / execute_command_sync auto-starts a stopped sandbox and wait for it to reach Running, and no sandbox status polling is required.
Running Commands
Pick the mode by workload:
| Mode | Method | Use for |
|---|---|---|
| Synchronous | execute_command_sync | Short commands where you only need the final stdout/stderr/exit code |
| Streaming | execute_command | Long-running or interactive commands where you consume output as it is produced |
Streaming execute_command (the CommandProcess handle) is currently available only in the Python SDK; execute_command_sync is available in all SDKs.
cmd runs directly (not through a shell), so to use shell features invoke /bin/bash -c "..." explicitly via args.
Synchronous
from google.protobuf.duration_pb2 import Duration
resp = w.sandbox.execute_command_sync(
"sandboxes/my-sandbox",
"/bin/bash",
args=["-c", "echo hello && whoami"],
execution_timeout=Duration(seconds=30),
)
print(resp.status, resp.exit_code, resp.truncated, resp.stdout, resp.stderr)
# Check resp.status for the terminal status: exit_code is unset when the command is
# killed (e.g. timed out) or fails to start. resp.truncated is True when the output
# hit the sync size cap - use execute_command (streaming) for large output.
Streaming
execute_command returns a CommandProcess handle once the command has started. Iterate stdout / stderr for live output and call wait() for the exit code; use it as a context manager to release the client-side stream and reader thread on exit. Closing the handle (or leaving the with block) tears down the client side only - the command keeps running in the sandbox. There is no per-command cancel API; to actually stop it, read its server-side PID from proc.pid and kill it with a separate command, e.g. w.sandbox.execute_command_sync(name, "/bin/kill", args=[str(proc.pid)]).
with w.sandbox.execute_command(
"sandboxes/my-sandbox",
"/bin/bash",
args=["-c", "for i in 1 2 3; do echo line $i; sleep 1; done"],
) as proc:
for line in proc.stdout:
print(line, end="")
exit_code = proc.wait() # also available as proc.exit_code once finished
stdout / stderr default to StreamMode.PIPE (buffered for reading); pass StreamMode.DEVNULL to discard, or StreamMode.STDOUT on stderr to merge it into stdout. Pass text=False for bytes instead of decoded str. Reattach to a still-running command with connect_command(name, command_id), and review history with list_commands(parent="sandboxes/my-sandbox").
Files
Manage the sandbox filesystem without a shell. All methods take parent (the sandbox name) plus a path. Write under the sandbox user's home directory, /home/sandbox-agent (the only persisted, writable tree; /home itself is root-owned, and paths outside the home directory are cleared on stop).
w.sandbox.create_directory(parent="sandboxes/my-sandbox", path="/home/sandbox-agent/data", parents=True)
for entry in w.sandbox.list_directory_contents(parent="sandboxes/my-sandbox", path="/home/sandbox-agent/data"):
print(entry.path)
w.sandbox.get_metadata(parent="sandboxes/my-sandbox", path="/home/sandbox-agent/data")
w.sandbox.rename_path(parent="sandboxes/my-sandbox", source="/home/sandbox-agent/a", destination="/home/sandbox-agent/b")
w.sandbox.delete_path(parent="sandboxes/my-sandbox", path="/home/sandbox-agent/b", recursive=True)
Use the sandbox execute_command APIs to move data in and out; native API/SDK support for uploading and downloading file contents (local <-> sandbox) is not yet available.
When to Use
- Agent / untrusted code execution - run model-generated code off the host, in isolation.
- Ephemeral build/test environments - create, use, delete.
Serverless Migration API
FIRST: Use the parent databricks-core skill for CLI installation, authentication, and profile selection.
Use the Compute Migrations API through the Databricks CLI to migrate a job to serverless. The service analyzes the job, backs it up, builds a serverless working copy, and optionally runs it in the background. Your part is to track the migration, review the working copy, present the changes to the user, and carry out their decision. Applying the changes to the original job requires the user's explicit approval.
When to Use This Skill
Use it when the user asks you to:
- migrate one job or many jobs to serverless,
- write a script that starts and tracks migrations,
- check on, approve, reject, or cancel migrations already in progress, or
- find the backup a migration took of a job.
Use databricks-serverless-migration for direct code edits, notebooks, pipelines, JAR tasks, or a job this service cannot migrate. For job definitions maintained in source control, read Jobs managed from source.
See the migration guide for supported workloads and limits. A successful create starts the migration; it does not establish that the workload is compatible. Incompatibilities can be discovered during planning or working-copy creation.
Migrate One Job
- Check the prerequisites once per workspace.
- Confirm the job and whether to set
auto_approveorrun_validation(see Settings chosen at creation). - Start the migration with
create, and record its migration ID andrequest_id. - Track progress until it reaches a review gate or a terminal state.
- At each review gate, review the working copy, present your findings to the user, and approve or reject as the user decides.
- After
MIGRATION_COMPLETE, verify the original job and report the result and the backup folder.
For a list of jobs, use Bulk migration to repeat this workflow and track each job's decisions.
Prerequisites
- CLI v1.24.0 or newer: check
databricks --versionanddatabricks compute-migrations --help. The command group and the verbs below must be available. If a command is missing, upgrade usingdatabricks-core. - Compute Agent preview enabled in the workspace. A workspace admin enables it under Previews. Without it, every call fails with
PERMISSION_DENIED("Migration APIs are not enabled for this workspace"). - CAN_MANAGE on each job you migrate.
The CLI help is generated from the API and is authoritative for command names, arguments, and flags. Run databricks compute-migrations <command> --help before first use of each command. Pass request bodies with --json and read responses with -o json.
Lifecycle
Prepare working copy
→ TRANSLATION_READY (skipped when auto_approve=true)
→ Optional validation run (run_validation=true)
→ MIGRATION_REVIEW (execution approval required)
→ Apply changes → MIGRATION_COMPLETE
- Planning. The service reads the job, its notebooks and files, and its cluster configuration, and works out the serverless changes: compute, dependencies, Spark configs, and code changes. The working copy is the first thing you review.
- Backup and working copy. The service saves a backup of the job and its workspace files (see Backups), then creates a working copy: a new job named
[Serverless Migration - <timestamp>] <job name>, taggedserverless_migration_source_job_id=<original job id>, running copies of the job's notebooks and files. The original job is not modified until you approve execution. - Gate 1,
TRANSLATION_READY. Skipped whenauto_approve=true. Otherwise, review the working copy. Approve to continue, or reject with a reason to request changes. - Validation run (only with
run_validation). Databricks runs the working-copy job once. The outcome is recorded invalidation_report. A failed run does not stop the migration: it moves toMIGRATION_REVIEWwithpassed: false. - Gate 2,
MIGRATION_REVIEW. Final approval to update the original job. Approving execution starts applying the working copy's settings and file contents to the original job, keeping its job ID, schedule, and permissions. - Complete. Verify the original job (see After completion).
Use either rejection command to request changes, with a specific rejection_reason. The same migration returns to planning and revises its working copy. Read migration_report again and review the current copy before approving it. Revisions are limited, and repeated rejections eventually fail the migration. When the user wants to stop the migration, use cancel-migration (see Cancellation).
Settings Chosen at Creation
auto_approve and run_validation are fixed when the migration is created and cannot be changed. Both default to false. Set either one only when the user asks for it.
| Field | Effect | Risk |
|---|---|---|
auto_approve | Skips gate 1 (TRANSLATION_READY). Gate 2 still waits for a decision. | With run_validation, the working copy runs before the user reviews it. |
run_validation | Runs the working-copy job once before gate 2. | The run is real: see below. |
A validation run is not sandboxed. The working copy runs in the same workspace against the same tables, volumes, checkpoints, and external systems as the original job, with the agent's changes applied. If the job is not idempotent, or has side effects, the validation run produces them. Examples: overwriting or appending to production tables, merges, sending emails or notifications, calling external APIs, triggering downstream jobs, or advancing streaming offsets and checkpoints. Enable run_validation only after the user confirms that one extra run of this job, right now, is safe, for example because the job is idempotent or writes only to test outputs. Never turn it on across a batch of jobs without that confirmation for each job.
Commands
| Operation | Command | Notes |
|---|---|---|
| Start | create --json '<migration>' --request-id <uuid> | asset_type is JOB; asset_uri is jobs/<job_id>. |
| Read one | get <migration_id> | Current state, reports, etag. |
| Read many | list | Returns a JSON array with -o json and paginates automatically. Filter the returned state values locally. Eventually consistent: confirm with get before acting. |
| Gate 1 | approve-migration-translation <migration_id> / reject-migration-translation <migration_id> <rejection_reason> | Reject takes the reason as a positional argument or as rejection_reason in --json. Approve accepts optional --additional-feedback. |
| Gate 2 | approve-migration-execution <migration_id> / reject-migration-execution <migration_id> <rejection_reason> | Same arguments and flags as gate 1. |
| Stop | cancel-migration <migration_id> | Any non-terminal state. Optional cancellation_reason. Does not undo applied changes; see Cancellation. |
| Remove record | delete <migration_id> | Hides the record and frees the job for a new migration. Rolls nothing back and cleans nothing up. Use cancel-migration to stop an in-flight migration. |
All commands are under databricks compute-migrations. Start a migration:
databricks compute-migrations create -o json --request-id "<uuid>" --json '{
"asset_type": "JOB",
"asset_uri": "jobs/<job_id>",
"auto_approve": false,
"run_validation": false
}'
Generate a fresh request_id per job and reuse it when retrying that job's create after a timeout or network error. The server returns the existing migration instead of creating a duplicate.
Pass the etag from your latest get on every approve, reject, and cancel, so the call fails rather than acting on a state you have not seen:
databricks compute-migrations reject-migration-translation <migration_id> -o json --json '{
"etag": "<etag>",
"rejection_reason": "Keep the existing output table names; task ingest writes to a different table in the working copy."
}'
| Error | Meaning | Action |
|---|---|---|
ALREADY_EXISTS | The job has a migration in progress, or one that already migrated it. | Find and confirm the existing migration as described below. |
INVALID_PARAMETER_VALUE | Bad request, or the job cannot be migrated. | Read the message; do not retry unchanged. |
PERMISSION_DENIED | No CAN_MANAGE on the job, or the preview is not enabled. | Report to the user. |
FAILED_PRECONDITION | Gate call made in the wrong state. | get, then act on the current state. |
ABORTED | The etag is stale. | get, re-check the state, retry with the new etag. |
After ALREADY_EXISTS, take the migration ID from the error and confirm its asset_uri and state with get. If the existing migration is MIGRATION_COMPLETE, report the job as already migrated. Do not delete the record and create another migration unless the user explicitly requests it.
Tracking Progress
Migrations take minutes to hours: planning and building the working copy usually take several minutes, and a validation run takes as long as the job. Use your harness's monitoring capabilities to track progress. get returns the current migration state; for many migrations, use list and confirm each one with get before acting.
| State | What to do |
|---|---|
PLANNING_PENDING, PLANNING_RUNNING, REVISION_PENDING | Planning is in progress. |
TRANSLATION_PENDING, TRANSLATION_CREATING | Working-copy creation is in progress. |
VALIDATION_SCHEDULING, VALIDATION_RUNNING | Validation is in progress. |
MIGRATION_PENDING, MIGRATION_RUNNING, MIGRATION_OBSERVATION | Migration is in progress. |
TRANSLATION_READY | Review the working copy, present it to the user, then approve or reject gate 1. |
MIGRATION_REVIEW | Review the working copy and validation_report, present them, then approve or reject gate 2. |
MIGRATION_COMPLETE | Verify the original job. |
NOT_SUPPORTED | The job cannot be migrated automatically. Report the reason from state_history, and offer databricks-serverless-migration for a manual migration. |
FAILED | Report the reason from state_history. If the migration failed during MIGRATION_RUNNING, check whether the original job was partly changed (see Backups). |
CANCELLED | Report the cancellation and check for applied changes or active validation runs as described in Cancellation. |
| Any other state | Read the migration again with get. If the state is still unrecognized, report it and stop automatic progress handling; do not assume it is in progress. |
state_history lists every state entered, with transition_time, the reviewer's user_comment, and the system's internal_comment (for example, the reason for a failure).
Cancellation
Cancellation does not undo changes already applied to the original job or files, or side effects of validation runs. An operation already in progress may finish after the migration reports CANCELLED; that state alone does not confirm that all work has stopped.
If execution was approved, or state_history includes MIGRATION_PENDING or MIGRATION_RUNNING, inspect the original job and files against the saved review record and backup. Report which changes are present and anything you cannot verify. Check active working-copy runs separately with databricks jobs list-runs --job-id <working_copy_job_id> --active-only -o json. Retain the recovery artifacts if the result is unverified or work may still be active.
Reviewing a Working Copy
The API returns links to the working copy, not a diff, so build the comparison yourself:
getthe migration.migration_report.primary_asset.idis the working-copy job's ID.- Compare job settings:
databricks jobs get <original_job_id> -o jsonagainstdatabricks jobs get <working_copy_job_id> -o json. Expect classic compute (job_clusters,new_cluster,existing_cluster_id) to be replaced by serverless settings (environmentsandenvironment_key), libraries to move into environment dependencies, unsupported Spark configs and init scripts to be removed, and possibly a performance mode to be set. The job name prefix, theserverless_migration_source_job_idtag, and task paths pointing at the copied notebooks are expected differences. - Compare code. The working-copy job's task settings name the notebooks and files it runs. Match its tasks to the original job's by
task_key, and take each task's path (notebook_task.notebook_pathorspark_python_task.python_file) from both jobs. Forfor_each_task, inspect its innertasktoo. Export each to a separate local file withdatabricks workspace export <path> --format SOURCE --file <local_file>and diff them. Also inspect utility notebooks invoked by the task notebooks for changes, such as through%runordbutils.notebook.run. - If a validation run happened, report
validation_report.passed,run_status, andrun_url. For a failed run, inspect it withdatabricks jobs get-run <run_id> -o json. Take the run ID fromrun_url. - Summarize the changes per task, including which tasks will migrate and which will stay on classic. Flag anything beyond a compatibility fix: changed business logic, different output tables or paths, removed steps, or dropped dependencies.
- Before approving execution, save the reviewed job settings and exported code, including utility notebooks, in a local folder for this migration ID. Include the original task paths, the working-copy-to-original path mapping, and the tasks approved for migration or intentionally kept on classic. This review record is needed after the working copy is deleted. Refresh it after any revision and review.
- Get the user's decision. A rejection reason should name the task and the specific change to make.
Rules while a migration is in progress:
- Approve gate 2 only with explicit user approval covering that job ID. A single decision may approve an explicitly identified set of jobs; the approval must cover the reviewed changes for each job.
- Do not edit the original job. Approving execution replaces its settings and files with the working copy's, discarding later edits.
- Do not edit the working copy to change the outcome. Reject with a reason instead.
After Completion
When the API reports MIGRATION_COMPLETE, verify the original job before reporting success:
- Read
databricks jobs get <job_id> -o jsonand compare it with the saved review record. Tasks approved for migration must have the reviewed serverless settings; tasks intentionally left on classic must retain their reviewed configuration. Account for the original job name and task paths, and removal of the working-copy tag. - Export the original task notebooks and files, including those used by nested tasks, along with the reviewed utility notebooks. Compare them with the saved reviewed code.
If approved changes are missing or differ from the review record, report the mismatch and offer to restore from the backup. Before retrying, restore any job settings or files changed by this migration from the pre-migration backup, with the user's approval. Then delete the migration (which frees the job) and create a new one. If the review record is unavailable, explain which parts of the result you could verify and which remain unverified.
The working-copy job or its copied files may remain after a successful migration. Report any leftovers and offer cleanup.
Report the migration ID, final state, original job, what changed, tasks left on classic, and the backup folder.
Backups
Before building the working copy, the service writes a backup to the workspace home of the identity that created the migration.
Files that tasks load from a Git repository are not copied; they stay in the repository. A new timestamped backup folder is written each time the service builds a working copy. Migrations never delete backups. Give the user the backup folder for every migration that reached TRANSLATION_READY or later.
There is no rollback command. For the backup location and file layout, restore instructions, and working-copy cleanup, follow Backups and rollback.
Bulk Migration
For multiple jobs, follow Bulk migration. It applies the single-job workflow to each migration, with separate tracking, settings, review records, and approval decisions.
Jobs Managed from Source
A migration changes the job in the workspace. If the job's definition comes from source, the next deploy reverts the migration. This covers jobs deployed from a Declarative Automation Bundle (settings.deployment.kind is BUNDLE) and jobs managed by Terraform or CI. For these jobs, tell the user before creating a migration. Either migrate the source (databricks-serverless-migration, databricks-dabs), or use the reviewed working copy as the reference for changes to make in the source.
Documentation
- Serverless migration guide, including the migration agent: https://docs.databricks.com/en/compute/serverless/migration
- Serverless compute limitations: https://docs.databricks.com/en/compute/serverless/limitations
- Serverless jobs: https://docs.databricks.com/en/jobs/run-serverless-jobs
Serverless Compute Migration
FIRST: Use the parent databricks-core skill for CLI basics, authentication, and profile selection.
Analyze existing Databricks code for serverless compute compatibility and guide migration from classic clusters. The skill follows a 4-step migration lifecycle: Ingest the workload → Analyze for compatibility → Test via A/B comparison → Validate and iterate.
Prefer the Compute Migrations API for Jobs
To migrate a job, first check whether databricks-serverless-migration-api is installed and its prerequisites are available. If so, use it to start and manage the migration. The service backs up the job, builds and optionally runs a serverless working copy, and changes the original only after review. Use the manual workflow below when:
- the API skill,
databricks compute-migrationsCLI commands, or Compute Agent preview are not available, - the workload is a standalone notebook or pipeline,
- the service reports the job as not supported, or
- the user wants you to make the changes yourself.
When to Use This Skill
- Migrating notebooks, jobs, or pipelines from classic compute to serverless
- Checking if existing code is serverless-compatible
- Writing new code that targets serverless compute
- Troubleshooting serverless-specific errors after migration
- Choosing between Performance-Optimized and Standard mode
Where to Run This Skill
Runs in any agentskills.io client: on a laptop (Claude Code, Cursor, ...; the default, install via databricks aitools install), or inside a Databricks workspace via Genie Code Agent mode (per-user or workspace-wide .assistant/skills/ folder). See Install in Databricks Genie Code for the three install methods and the serverless-compute caveat (the Databricks CLI isn't pre-installed on serverless, so some deploy steps need adjustment).
If you finish a migration analysis for a user who's currently running you from a laptop client, mention the Genie Code option once at the end — many users prefer iterating on migrations inside the workspace where the workload lives.
Understanding Migration Blockers
Migration blockers fall into three categories. Focus your effort on category 2 — that's where this skill helps most.
| Category | Description | Action |
|---|---|---|
| 1. Feature expanding | Databricks is actively expanding support (e.g., SparkML, custom JDBC) | Use the workaround now and revisit later |
| 2. Code/config change needed | Your code uses patterns that need updating for serverless (e.g., RDDs, DBFS, streaming triggers) | This skill helps here — it detects these patterns and provides fixes |
| 3. Classic-only | Workload requires capabilities not available on serverless (e.g., root OS access, R language) | Keep on classic compute |
Decision Tree: Is My Workload Ready?
Workload → Check language
├── R code → Category 3: keep on classic
├── Scala notebook cells → Category 2: port to PySpark/SQL, or compile as a JAR job task
├── Compiled Scala JAR (spark_jar_task, build.sbt/pom.xml/build.gradle) → see references/jar-migration.md
│ ├── Scala 2.12 build? → recompile against 2.13.16
│ ├── Bundles spark-core/spark-sql? → use databricks-connect % Provided
│ └── Bundles libs on the kernel classpath? → mark % Provided (see the classpath table in references/jar-migration.md)
├── Python / SQL → Continue
├── Uses RDD APIs? → Category 2: rewrite to DataFrame API (see fixes below)
├── Uses DBFS paths? → Category 2: migrate to UC Volumes
├── Uses Hive Metastore? → Category 2: migrate to Unity Catalog (or use HMS Federation)
├── Uses df.cache/persist? → Category 1: remove and materialize to Delta (native support coming soon)
├── Uses streaming?
│ ├── ProcessingTime trigger → Category 2: use AvailableNow or migrate to SDP
│ ├── Continuous trigger → Category 2: use SDP continuous mode
│ ├── No trigger specified → Category 2: add explicit .trigger(availableNow=True)
│ └── AvailableNow / Once → Ready ✓
├── Uses init scripts? → Category 2: use Environments
├── Uses VPC peering? → Category 2: use NCCs / Private Link
├── Uses unsupported Spark configs? → Category 2: remove (serverless auto-tunes)
├── Uses custom JDBC drivers? → Category 2: use Lakehouse Federation or built-in JDBC
├── Uses Docker containers? → Category 3: use Environments for libs, or keep on classic
└── All clear → Ready for serverless ✓
Migration Workflow
Step 1: Ingest — Gather Workload Context
Confirm the migration target is serverless compute. This skill is purpose-built for classic → serverless migrations. The checks, fixes, and workflow all target the serverless compute architecture (Spark Connect, Environments, NCCs). If the user wants to upgrade between classic DBR versions instead, this skill does not apply — classic DBR upgrades have a different compatibility surface and should follow the standard DBR upgrade guide.
Collect the full picture of what needs to migrate to serverless:
- Read the user's notebook/script files
- Identify the classic cluster configuration (instance type, DBR version, Spark configs, init scripts, libraries)
- Note the networking setup (VPC peering, instance profiles, mounts)
- Understand the workload type: batch job, streaming, interactive notebook, pipeline
- Determine the target: the output is always a serverless compute configuration, not a classic cluster with a newer DBR
Step 1.5: Handle Multi-Notebook Workloads
If the workload spans 3+ user notebooks (exclude _resources/ setup notebooks) or any notebook exceeds ~5KB of source, process notebooks one at a time to avoid context overflow (autocompact thrashing): enumerate first (including upstream sources declared in bundle_config.py, H4), then per notebook read → run the Step 2 checklist → record a structured summary and drop the source, then synthesize the plan from the summaries. Trigger the Failure Reporting Protocol once per workload, not per notebook. Don't merge all notebooks into one file or act on all of them in one mega-turn. Full procedure (summary fields, scratch-dir layout, report aggregation): Multi-Notebook Workloads.
Step 2: Analyze — Scan for Serverless Readiness
Read notebooks before running them — do not rely on failed job runs to discover issues. A pre-run scan surfaces incompatibilities faster than iterating on error traces, and many serverless failures (hardcoded catalog references, init scripts, missing dependencies) are easy to spot statically but expensive to debug after a failed run.
Before creating or running any test job:
- Read every notebook and source file referenced by the job
- Scan for all hardcoded catalog/schema references (e.g.,
spark.table("main.schema.table"),spark.sql("... FROM main..."),catalog = "main") - Check for dependency patterns: init scripts, local wheel files, custom install functions,
%pip installlines - Locate any
requirements.txtor equivalent and resolve the full dependency set - Flag OS-level installs (
apt install,yum install) for conversion or escalation
Scan the code for patterns that are incompatible with the serverless compute architecture. These checks are serverless-specific — most of these patterns work fine on classic compute regardless of DBR version. For each issue found, report:
- Category: Which of the 3 blocker categories it falls into
- Severity: Blocker (must fix for serverless) / Warning (should fix) / Info (awareness)
- Pattern: What was detected and where
- Fix: Specific remediation targeting serverless compute
Cell-magic boundary lint (A1), high impact: in any cell with a # MAGIC %<word> line, every non-blank line must start with # MAGIC . A plain-Python comment above # MAGIC %run in the same cell makes %run fall back to IPython (File './…' not found); put migration notes in their own # MAGIC %md cell instead.
Pattern catalog: load Analyze Checklist for every Step 2 scan. It lists each check with severity and fix: A APIs (RDD, caching, MLflow/AutoML A2–A5, M1–M4), B data access (B1–B2), C streaming, D config (D1), E libraries (E1–E2), F networking, G sizing, H job-level config (H1–H4).
Required Output: Serverless Environment Specification
The migration output MUST include a Serverless Environment specification alongside migrated code. Generate this by:
- Scanning all
importstatements and%pip installlines to detect required packages - Extracting init script
pip installcommands from the job configuration - Producing a JSON block suitable for the Jobs API
environmentsfield:
{
"environment_key": "Default",
"spec": {
"client": "2",
"dependencies": ["mlflow==2.12.1", "scikit-learn==1.3.0", "xgboost==2.0.3"]
}
}
Important: ML runtime is NOT available on serverless and ML libraries (mlflow, scikit-learn, hyperopt, xgboost, tensorflow, torch, etc.) are NOT pre-installed, so list them explicitly in the environment spec dependencies.
Step 3: Test — Two-Branch Strategy
Use separate branches for testing and production to keep test-only workarounds out of the code that ships. The test branch is a safe sandbox for experimentation; the production branch contains only changes that production actually needs.
Branch-by-branch comparison (names, base branch, PR, catalog, what each may contain): Testing Strategy.
Test branch (serverless-test-{job_name}-{timestamp}): Temporary, no PR needed.
- Create a branch from your current working branch
- Set up test data: create sampled copies of upstream tables in a test catalog using job lineage (see Testing Strategy § Test Data Setup)
- Parameterize the catalog so the notebook works with both test and production data (see Quick Fixes § Parameterize catalogs)
- Apply all compatibility fixes discovered in Step 2
- Create a serverless test job and run it
- If it fails, get the error output, debug, fix, and retry
- Document which changes are test workarounds vs. real compatibility fixes
Production branch (serverless-prod-{job_name}): PR required, created from master.
- Create a new branch from master (NOT from the test branch)
- Apply ONLY the real compatibility fixes — no test workarounds
- Apply job config changes (see Quick Fixes and Debugging)
- Commit and create a PR
Decision Tree: Should This Change Go to Production?
| Change type | Production? | Reason |
|---|---|---|
| Remove incompatible Spark configs | Yes | Serverless compatibility fix |
| Update library versions | Yes | Serverless compatibility fix |
| Replace DBFS paths with UC Volumes | Yes | Serverless compatibility fix |
| Remove init scripts, add Environments | Yes | Serverless compatibility fix |
| Fix hardcoded cluster settings | Yes | Serverless compatibility fix |
| Catalog override to test catalog | No | Test workaround only |
| Empty DataFrame handling for missing test data | No | Test workaround only |
| Date range limiting for faster tests | No | Test workaround only |
Simple test: Would production fail without this change on serverless? If yes → include. If no → test branch only.
A/B Comparison
After both branches are ready, compare classic vs serverless outputs (row count, schema, exceptAll row diff; snippet and the temporary bridge-config rule in Testing Strategy § A/B Comparison). Test against sampled copies of upstream tables in a test catalog, never production data (Test Data Setup).
Step 4: Validate — Confirm and Monitor
Once the A/B comparison passes:
- Merge the production branch PR
- Switch the production job to serverless compute
- Monitor cost via system tables (
system.billing.usage) and budget policies - Remove any temporary bridge configurations
- Set up budget alerts for cost visibility
Migration Deliverables
Surface all six at the end of a successful run: test branch name/URL, production branch name/URL, test job ID and run URL, classic vs serverless comparison, serverless environment spec, and change summary (production vs test-only, with reasons). What each is and why it matters: Testing Strategy § Migration Deliverables. If any deliverable is missing, the migration is incomplete — do not mark it as done.
Stopping Conditions
Do not attempt workarounds for these — surface them to the user and stop:
- Permission failures on source tables, the test catalog, or the workspace
- Category 3 blockers (R code, custom Spark data source JARs, features that require classic compute)
- SQL warehouse or test catalog not available
- Repeated failures (typically 5+) with no new information in the error trace — generate a failure report instead (see Failure Reporting Protocol)
Quick Fixes and Debugging
- Quick Fixes: before/after snippets for DBFS → UC Volumes, RDD → DataFrame, streaming triggers (+ OOM caps for Delta, Kafka, Auto Loader), caching removal, runtime serverless detection (
IS_SERVERLESS), and catalog parameterization for tests. - Debugging Errors: error message → fix table for failed serverless runs. Always get the actual error with
w.jobs.get_run_output(run_id=...)before guessing.
Transform job config from classic to serverless
Remove job_clusters/new_cluster, add environments with serverless spec, replace job_cluster_key with environment_key, remove init_scripts. Full rules (init scripts → dependencies, Maven/JAR libraries, spark_conf triage) and before/after JSON: Configuration Guide.
Environment version mapping (match to the DBR version the workload was on):
| Classic DBR | Serverless spec.client | Python |
|---|---|---|
| 13.x, 14.x | "1" | 3.10 |
| 15.x | "2" | 3.11 |
| 16.x+ | "3" | 3.12 |
Performance Mode Selection
Performance-Optimized (default in UI and API; <50 s startup; higher cost; notebooks, jobs, SDP) for interactive, dev, and time-sensitive work. Standard (4-6 min startup; significantly lower cost; jobs and SDP only, must be selected explicitly) for batch ETL and scheduled pipelines. Standard mode is NOT available for notebooks. Selection guide and how to set it: Configuration Guide § Performance Mode Selection.
Serverless Defaults to Know
| Setting | Value |
|---|---|
| REPL VM memory | 8GB default (high-memory option available) |
| Max executors | 32 (Premium), 64 (Enterprise) — raise via support |
| Supported Spark configs | 6 only: spark.sql.shuffle.partitions, spark.sql.session.timeZone, spark.sql.ansi.enabled, spark.sql.files.maxPartitionBytes, spark.sql.legacy.timeParserPolicy, spark.databricks.execution.timeout |
| Debugging | Query Profile (no Spark UI) |
| ANSI SQL | Enabled by default (configurable) |
Failure Reporting Protocol
When migration fails or hits an unmigratable pattern, generate a structured failure report and offer the user a one-step path to file it as a GitHub issue. This feedback loop is how the skill learns — without it, gaps go undiscovered.
The report schema, MUST-NOT-contain rules, anonymization regexes, deterministic post-serialization scrub, output template, and URL recipe are in Failure Reporting; load it before writing a report.
Decision tree: when to offer to file an issue
Offer to file a GitHub issue any time the workload cannot be fully migrated to serverless as-is. Reports from "known" patterns (R, Scala, custom JAR data sources, JVM access, third-party connectors) are just as valuable as reports from unknown patterns — they tell maintainers which gaps users hit most often, which drives prioritization.
Concretely, ALWAYS offer if any of these is true:
- Workload contains any Category 3 (classic-only) blocker — R, Scala notebook cells, custom JAR data sources, JVM/Py4J access, third-party connectors without serverless equivalents, native binary dependencies, etc. The fact that the pattern is "documented as Cat 3" is not a reason to skip the offer.
- Retries exhausted —
retry_count >= max_retries(typically 5) and final status is FAILED - Unknown pattern — a classic-compute construct was detected that isn't in the skill's catalog
- Fix didn't resolve — a known fix was applied but the workload still fails on serverless
- Explicit request — the user invokes
/migration-report
Do NOT offer to file only when:
- The migration succeeded fully (even after retries), or
- The workload is already serverless-compatible and required no changes.
Rules that always apply: write the report locally to ~/.databricks-migration-skill/reports/failure-<ISO-timestamp>.json (pattern IDs and notebook_characteristics only, no code, paths, error text, identifiers, or credentials). Default output is local-only: show the path and ask the user to review. Produce the pre-filled GitHub URL or gh issue create command only in a later turn, after the user explicitly confirms and the deterministic scrub of the final JSON is clean. Never transmit the report automatically; if the user declines, log the local path and move on.
Success report
When migration succeeds, generate a per-workload report with the same structure as the failure report (minus the redaction-to-publish step), listing the patterns detected and the fixes applied for each workload. Write to ~/.databricks-migration-skill/reports/success-<timestamp>.json and tell the user the file path.
Multiple workload migration
When several independent workloads are in scope, migrate them in parallel (e.g. one subagent per workload). Keep the per-notebook steps within a single workload sequential, as in Step 1.5.
Reference Guides
- Analyze Checklist: Step 2 pattern catalog (A–H, with IDs) and A1 lint
- Quick Fixes: Before/after snippets for the most common fixes
- Debugging Errors: Error → fix table for failed serverless runs
- Multi-Notebook Workloads: Step 1.5 one-notebook-at-a-time procedure
- Testing Strategy: Branch table, test data, A/B snippet, deliverables
- Compatibility Checks: Detection details and regexes for 40+ checks
- Streaming Migration: Trigger migration, SDP continuous mode, continuous jobs
- Networking and Security: VPC peering to NCCs, Private Link, firewall setup
- Code Patterns: Before/after code for every pattern
- Configuration Guide: Spark configs, Environments, budget policies, job JSON
- MLflow on UC: AutoML → sklearn, UC registration,
spark_udffix - Multi-source enumeration:
bundle_config.pyupstream sources - Failure Reporting: Report schema, redaction, scrub, publish flow
- Install in Databricks Genie Code: Run inside a workspace
- JAR Migration: Scala JAR tasks: 2.13.16, Connect, classpath conflicts
Documentation
- Serverless compute overview: https://docs.databricks.com/compute/serverless/
- Migration guide: https://docs.databricks.com/compute/serverless/migration
- Limitations: https://docs.databricks.com/compute/serverless/limitations
- Best practices: https://docs.databricks.com/compute/serverless/best-practices
- Spark Connect vs. classic: https://docs.databricks.com/spark/connect-vs-classic
- More (notebooks, jobs, SDP, UC upgrade, Spark configs): Configuration Guide
Deploying Databricks jobs and pipelines on serverless
FIRST: Use the parent databricks-core skill for CLI basics, authentication and profile selection.
This skill decides compute. Where another skill or example (for example databricks-jobs) shows
cluster configuration, follow this skill instead.
Serverless is the default compute for every job, pipeline and bundle you create. If the user explicitly asks for a cluster, do what they ask and mention the serverless option in one line. When editing an existing job, keep its compute unless the user asks to change it; tasks you add follow this skill.
1. Serverless or classic
Use classic compute only for:
- R code, or Scala in notebook cells (
%scala): serverless notebooks run Python and SQL only; - custom Docker images;
- OS packages (
apt-get) or native libraries with no pip equivalent; - custom Spark data source JARs;
- a stated hard requirement for an instance type or Databricks Runtime version;
- an explicit user request;
- existing code over the budget in section 5;
- a workspace without serverless: if creating or running the job fails because serverless is not available there, put the whole job on classic and say so.
Then give only the blocked task a cluster (one small job_clusters entry with the latest LTS
runtime and autoscale: {min_workers: 1, max_workers: 4}), keep every other task serverless, and
state the blocker in one line. For a blocked pipeline, give the pipeline a clusters block instead
of serverless: true.
Not reasons for classic: Scala or Java in a JAR task (JAR tasks run on serverless), GPUs (serverless GPU), large data, long runtimes, Kafka, cost (serverless STANDARD is on par with or cheaper than on-demand classic in most cases), or habit.
2. Put every task on serverless
- New jobs: leave out
new_cluster,job_clusters,job_cluster_key,existing_cluster_id,instance_pool_id,node_type_id,num_workers,autoscale,spark_version. A task without a cluster runs on serverless. In an existing job, keep the cluster settings it has. - Notebook tasks need nothing else. Python script, wheel and JAR tasks take their libraries
from a job-level environment. Use the latest environment version the workspace offers, as a
quoted string:
environments: - environment_key: default spec: environment_version: "4" # example; use the latest dependencies: ["requests==2.32.3"] # pin versions; JARs go in java_dependencies tasks: - task_key: main spark_python_task: {python_file: /Workspace/.../main.py} environment_key: default - SQL tasks: a serverless SQL warehouse. Pipelines:
serverless: true, noclustersblock. - Streaming:
.trigger(availableNow=True), the trigger serverless supports (noprocessingTimeorcontinuoustriggers). For always-on processing, run that stream in a continuous job (continuous: {pause_status: UNPAUSED}), which starts the next run as soon as one finishes, or use a continuous pipeline. - Tag every job you create for resource tracking, as other Databricks skills do: job-level
tags: {"aidevkit_project": "ai-dev-kit"}. Keep tags the user already has.
3. Choose the performance mode
Set the job-level performance_target:
STANDARDwhen nothing is time-critical and cost matters more than speed: scheduled batch and ETL, nightly or weekly runs, or any job where the user states no latency need. Runs start within minutes instead of seconds and cost less. This is the default choice.PERFORMANCE_OPTIMIZEDonly when the user names an SLA or a deadline that is tight relative to the runtime, a person waits on the result, or the job runs every 30 minutes or more often.
performance_target applies to the whole job, notebook tasks included (only interactive notebooks
always run performance-optimized). Say which mode you chose and why in one line, e.g. "STANDARD:
nightly batch, no deadline."
Serverless GPU tasks ignore performance_target.
Pipelines have no performance_target of their own. To schedule a pipeline, trigger it from a job
with a pipeline_task and set the mode on that job; the job's mode applies to the pipeline update.
A continuous pipeline runs in STANDARD only when a continuous job runs it.
4. Code you write runs on serverless from the start
Serverless reads Unity Catalog tables, /Volumes/... paths, cloud paths behind a Unity Catalog
external location, and the sample data under dbfs:/databricks-datasets/. It cannot read DBFS
mounts (dbfs:/mnt/...) or other DBFS root paths; only those count as a DBFS blocker.
DataFrame and SQL APIs only (no RDDs, sc.*, spark.sparkContext); Unity Catalog three-part table
names; /Volumes/... paths instead of DBFS mounts (dbfs:/mnt/...); availableNow streaming
triggers; no .cache() or .persist(); no spark.conf.set for executor, driver or memory
settings; pinned dependencies in the environment, never init scripts.
5. Existing code: a quick check, not a migration
The user is waiting for the deployment. Spend one short pass, then deploy.
-
Scan only the files the job runs (a text search such as
greporrg; do not read the whole repo) for:sc.,sparkContext,.rdd,parallelize,mapPartitions,reduceByKey,groupByKey,dbfs:/,/dbfs/,dbutils.fs.mount,.cache(,.persist(,spark.conf.set,processingTime,continuous=,writeStream,hive_metastore.,GLOBAL TEMP,hivevar,init_scripts,apt-get,docker, and other languages:%scalaor%ras the whole magic (# MAGIC %r, not%run),.rfiles and R notebooks,SparkRandsparklyr. R and Scala cells go to classic (section 1). -
Minor blockers: fix them and deploy on serverless if every fix is mechanical and local, keeps the logic, needs no new infrastructure, and there are about 5 small edits or fewer in total:
- delete
.cache(),.persist(),.unpersist(),setLogLevel, executor/driver/AQEspark.conf.setlines - move
pip installs orrequirements.txtinto the environment, pinned - add
.trigger(availableNow=True)to a stream that sets no trigger, in a job that runs on a schedule anyway (a stream with aprocessingTimeorcontinuoustrigger is a step 4 blocker) - replace a
dbfs:/mnt/...path with a/Volumes/...path only when that volume's storage location is the mount's source, so the job reads the same data; a volume with a similar name is not enough. Otherwise the mount is a step 4 blocker. CREATE GLOBAL TEMP VIEWbecomesCREATE TEMP VIEW, and itsglobal_temp.<view>reads become<view>, when the same file creates and reads the view (serverless has no global temp views); a global temp view that another task or notebook reads is a step 4 blocker- add the catalog to a two-part table name only when you know which Unity Catalog catalog holds the
table (for example the job's default catalog); a table in
hive_metastoreis a step 4 blocker
List every edit you made in your reply.
- delete
-
RDD or SparkContext code: translate it to DataFrames and deploy on serverless only if all of these hold:
- it is one file of about 200 lines or fewer;
- the change is a pure API translation: the same logic, expressed with DataFrame or SQL;
- every input path, table, output table and parameter stays exactly as it was. Never swap in another dataset, even a sample one;
- every input is already readable on serverless (section 4: a Unity Catalog table, an existing
/Volumes/...path, a cloud path behind an external location, ordbfs:/databricks-datasets/), so nothing new has to be set up.
Save the original next to it as
<name>.classic.pyand point the job at the translated file. Tell the user that the translation has not been run yet and suggest one test run that compares the output with the classic version. -
Anything bigger: do not rewrite. Put that task on classic now (section 1), keep the other tasks serverless, and end with: "Kept on classic because . To move it to serverless, use the databricks-serverless-migration skill." Bigger means: RDD code that fails any condition in step 3 (for example it reads a DBFS mount), mounts with no volume on the same storage, Hive Metastore to Unity Catalog, recompiling a JAR, networking changes, changing what a stream does (an always-on
processingTimeorcontinuousstream), or more than about 5 edits besides a step 3 translation.
Do not stop to ask whether to migrate. Deploy, then report what you did.
Common Issues
| Issue | Solution |
|---|---|
Libraries field is not supported for serverless task | Move libraries into environments[].spec.dependencies (JARs: java_dependencies) and set environment_key on the task |
| Serverless job starts too slowly | performance_target: PERFORMANCE_OPTIMIZED, only if the user needs fast starts (section 3) |
Only remote Spark sessions ... are supported or another RDD error | RDD/SparkContext code: section 5, step 3 or 4 |
[PATH_NOT_FOUND] or no access on dbfs:/mnt/... | DBFS mounts do not work on serverless: use a volume on the same storage (section 5, step 2), otherwise step 4 |
| Serverless is not enabled or not available in the workspace | Put the job on classic and tell the user (section 1) |
A sibling skill's example uses job_clusters for a new job | Leave the cluster block out; this skill decides compute. Existing jobs keep their compute |
Related Skills
- databricks-core - CLI basics, authentication, profiles
- databricks-jobs - job structure, task types, triggers, notifications
- databricks-pipelines - Lakeflow Spark Declarative Pipelines
- databricks-serverless-migration - migrating existing classic workloads to serverless
Documentation
Databricks Setup-Local
REQUIRED FIRST: Use databricks-core for CLI, authentication, and profile selection. Never use a default profile. For an existing environment, use databricks-execution-compute.
Workflow
1. Check CLI and authentication
The CLI must be >= v1.16.0 -- the release that added --no-dbconnect and the E_PROVISION_CONFLICT failure this skill routes. Compare deterministically -- never eyeball the version (1.9.0 is older than 1.16.0, and a lexical string compare gets this wrong). Run this gate first and do not proceed if it exits non-zero:
# Subshell so a failed gate reports non-zero without closing a persistent shell.
(
# Read stdout only -- upgrade nags on stderr carry their own semver and would win the parse.
if ! raw="$(databricks version 2>/dev/null)"; then
echo "STOP: databricks CLI is missing or failed to run -- install or upgrade it via databricks-core first"
exit 1
fi
have="$(printf '%s\n' "$raw" | grep -oE 'v[0-9]+\.[0-9]+\.[0-9]+' | head -n1 | tr -d v)"
[ -n "$have" ] || have="$(printf '%s\n' "$raw" | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -n1)"
if [ -z "$have" ]; then
echo "STOP: could not read a version from 'databricks version' -- upgrade via databricks-core first"
exit 1
fi
case "$raw" in
*-dev*)
echo "NOTE: dev build $have -- the floor cannot be checked; ask the user to confirm this build has setup-local"
;;
*)
if [ "$(printf '%s\n%s\n' "1.16.0" "$have" | sort -V | head -n1)" != "1.16.0" ]; then
echo "STOP: databricks CLI $have is older than v1.16.0, the floor this skill requires -- older CLIs can run setup-local but not as documented here; upgrade via databricks-core first"
exit 1
fi
;;
esac
)
Only after the gate passes, authenticate with the selected profile:
databricks auth describe --profile <PROFILE>
Prefer the latest stable CLI; no online lookup is required. If the gate exits non-zero (older than v1.16.0, missing CLI, or no readable version), or setup-local is absent from help, reports unknown command, or rejects a flag this skill tells you to pass, stop. If it prints NOTE: dev build, the floor is unverifiable -- confirm with the user before continuing. Use databricks-core to upgrade with approval and verify; never recreate setup-local manually.
Use the selected profile for every workspace command. Do not convert another package manager without approval.
2. Confirm the project directory
Confirm the root containing (or intended to contain) pyproject.toml, .venv, and uv.lock; ask if multiple roots are plausible. Use it for preview and apply. It must be greenfield or uv-managed, but need not be writable for preview.
3. Select one target
Choose exactly one branch:
- Cluster: use
--cluster-id <ID>or--cluster-name <NAME>. If unknown, list clusters with the selected profile and ask; see examples. - Serverless: use
--serverless-version <N>. No version-list command exists; ask if unspecified. - Job task: use
--job-task <JOB_ID>.<TASK_KEY>. If the task is unknown, rundatabricks jobs get <JOB_ID> --profile <PROFILE> --output json, present task keys, and ask. - Bundle: a project with
databricks.yml. Usedatabricks-dabsto inspect its root and selected target. Omit compute flags only when that target resolves supported classic or serverless compute; otherwise ask. Add--target <BUNDLE_TARGET>for a named target.
Never combine compute flags. If no branch resolves, ask the user.
4. Preview
Dry-run first; it writes and installs nothing:
databricks environments setup-local --profile <PROFILE> <TARGET_ARGS> --dry-run --output json
For bundles, <TARGET_ARGS> is empty or --target <BUNDLE_TARGET>. Default to normal mode. Use --no-dbconnect only when the user explicitly does not want this command managing databricks-connect; the result's mode still reports constraints-only for it. See JSON output and examples.
5. Obtain approval and apply
Before apply, verify the directory is writable and run uv --version. Ask before installing uv; never silently set DATABRICKS_LOCALENV_AUTO_INSTALL_UV=1 or run a remote installer.
Show the target, versions, warnings, plan.diff, and directory. Explain that apply may:
- back up and rewrite
pyproject.toml; - install Python and dependencies;
- update
.venvanduv.lock.
For --serverless-version N in a bundle, also disclose the post-apply job YAML synchronization described below so approval covers both mutations.
Apply only after the user requested provisioning or approves that plan for the named directory. Preserve the directory, profile, target, and mode.
If the directory, profile, target, mode, or project files change after preview, rerun --dry-run, show the new plan, and obtain approval again. Treat its resolved Python, databricks-connect, and managed constraints as authoritative; do not substitute guessed versions. Reconcile user-owned dependency conflicts separately, with approval.
6. Handle the result
ok: true: for--serverless-version Nin a bundle, update every existing jobenvironments[].spec.environment_versionin its YAML sources to"N", then validate the bundle. Report if none exist; do not invent one. Skip this for cluster and job-task targets. Report target, versions, warnings, andvenvPath; preferuv run <cmd>or derive the platform-specific interpreter fromvenvPath.ok: false: use troubleshooting. Ask before diagnostic runs that mutate files and before filing an external issue.- No JSON: inspect stderr for a pre-pipeline CLI, authentication, directory, or cache error.
Spark Structured Streaming
Production-ready streaming pipelines with Spark Structured Streaming. This skill provides navigation to detailed patterns and best practices.
Quick Start
from pyspark.sql.functions import col, from_json
# Basic Kafka to Delta streaming
df = (spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "broker:9092")
.option("subscribe", "topic")
.load()
.select(from_json(col("value").cast("string"), schema).alias("data"))
.select("data.*")
)
df.writeStream \
.format("delta") \
.outputMode("append") \
.option("checkpointLocation", "/Volumes/catalog/checkpoints/stream") \
.trigger(processingTime="30 seconds") \
.start("/delta/target_table")
Core Patterns
| Pattern | Description | Reference |
|---|---|---|
| Kafka Streaming | Kafka to Delta, Kafka to Kafka, Real-Time Mode | See references/kafka-streaming.md |
| Real-Time Mode (RTM) | Sub-second E2E latency — cluster setup, slot math, supported ops (incl. stream-stream inner join on DBR 18+), transformWithState, observability, error classes, delivery semantics | See references/real-time-mode.md |
| Lakebase Sink | Write streaming records into Lakebase Postgres with transactional upserts. Native format("postgresql") sink (DBR 18.3+) and manual foreach sink as a fallback | See references/lakebase-sink-python.md |
| Stream Joins | Stream-stream joins, stream-static joins | See references/stream-stream-joins.md, references/stream-static-joins.md |
| Multi-Sink Writes | Write to multiple tables, parallel merges | See references/multi-sink-writes.md |
| Merge Operations | MERGE performance, parallel merges, optimizations | See references/merge-operations.md |
Custom State (transformWithState) | Built-in stateful operator or transformWithState; timers (timeMode="ProcessingTime" never lets AvailableNow jobs finish), TTL, initial state, schema evolution, state reader, migrating from applyInPandasWithState | See references/transform-with-state.md |
Configuration
| Topic | Description | Reference |
|---|---|---|
| Checkpoints | Checkpoint management and best practices | See references/checkpoint-best-practices.md |
| Stateful Operations | Watermarks, state stores, RocksDB configuration | See references/stateful-operations.md |
| Trigger & Cost | Trigger selection, cost optimization, RTM | See references/trigger-and-cost-optimization.md |
Best Practices
| Topic | Description | Reference |
|---|---|---|
| Production Checklist | Comprehensive best practices | See references/streaming-best-practices.md |
Production Checklist
- Checkpoint location is persistent (UC volumes, not DBFS)
- Unique checkpoint per stream
- Fixed-size cluster (no autoscaling for streaming)
- Monitoring configured (input rate, lag, batch duration)
- Exactly-once verified (txnVersion/txnAppId)
- Watermark configured for stateful operations
- New custom state uses
transformWithStatewhere supported, withtimeMode="EventTime"or"None"in scheduled or serverless jobs - Left joins for stream-static (not inner)
Catalog and schema are always user-supplied — never default to any value. If the user hasn't provided them, ask. For any UC write, always create the schema if it doesn't exist before writing data.
Databricks Synthetic Data Generation
Generate realistic, story-driven synthetic data for Databricks using Spark + Faker + Pandas UDFs (strongly recommended).
Data Must Tell a Business Story
Synthetic data should demonstrate how Databricks helps solve real business problems.
The pattern: Something goes wrong → business impact ($) → analyze root cause → identify affected customers → fix and prevent.
Key principles:
- Problem → Impact → Analysis → Solution — Include an incident, anomaly, or issue that causes measurable business impact. The data lets you find the root cause and act on it.
- Industry-relevant but simple — Use domain terms (e.g., "SLA breach", "churn", "stockout") but keep the schema easy to understand. A few tables, clear relationships.
- Business metrics with $ impact — Revenue, MRR, cost, conversion rate. Every story needs a dollar sign to show why it matters.
- Tables explain each other — Ticket spike? Incident table shows the outage. Revenue drop? Churn table shows who left and why. All data connects.
- Actionable insights — Data should answer: What happened? Who's affected? How much did it cost? How do we prevent it?
Why no flat distributions: Uniform data has no story — no spikes, no anomalies, no cohort, no 20/80, no skew, nothing to investigate. It can't show Databricks' value for root cause analysis.
References
| When | Guide |
|---|---|
| User mentions ML model training or complex time patterns | references/1-data-patterns.md — ML-ready data, time multipliers, row coherence |
| Errors during generation | references/2-troubleshooting.md — Fixing common issues |
Critical Rules
- Data tells a story — Something goes wrong, impacts $, can be analyzed and fixed. Show Databricks value.
- All data serves the story — Every table and column must be coherent and usable in dashboards or ML models. No orphan data, no random noise — if it doesn't help explain or plot a futur dashboard or predict, don't generate it.
- Industry terms, simple schema — Use domain-specific vocabulary but keep it easy to understand (few tables, clear relationships)
- Never uniform distributions — Skewed categories, log-normal amounts, 80/20 patterns. Flat = no story = useless
- Enough data for trends — ~100K+ rows for main tables so patterns survive aggregation
- Ask for catalog/schema — Never default, always confirm before generating
- Present plan for approval — Show tables, distributions, assumptions before writing code
- Master tables first — Generate parent tables, write to Delta, then create children with valid FKs
- Use Spark + Faker + Pandas UDFs — Scalable, parallel. Polars only if user explicitly wants local + <30K rows
- Use Databricks Connect Serverless by default to generate data — Update databricks-connect on python 3.12 if required (avoid using execute_code unless instructed to not use Databricks Connect)
- No
.cache()or.persist()— Not supported on serverless. Write to Delta, read back for joins - No Python loops or
.collect()— Use Spark parallelism. No driver-side iteration, avoid Pandas↔Spark conversions
Generation Planning Workflow
Before generating any code, you MUST present a plan for user approval.
⚠️ MUST DO: Confirm Catalog Before Proceeding
You MUST explicitly ask the user which catalog to use. Do not assume or proceed without confirmation.
Example prompt to user:
"Which Unity Catalog should I use for this data?"
When presenting your plan, always show the selected catalog prominently:
📍 Output Location: catalog_name.schema_name
Volume: /Volumes/catalog_name/schema_name/raw_data/
This makes it easy for the user to spot and correct if needed.
Step 1: Gather Requirements
Ask the user about:
- Catalog/Schema — Which catalog to use?
- Domain — E-commerce, support tickets, IoT, financial? (Use industry terms)
If user doesn't specify a story: Propose one. Don't generate bland data — suggest an incident, anomaly, or trend that shows Databricks value (e.g., "I'll include a system outage that causes ticket spike and churn — this lets you demo root cause analysis").
Step 2: Present Plan with Story
Show a clear specification with the business story and your assumptions surfaced:
📍 Output Location: {user_catalog}.support_demo
Volume: /Volumes/{user_catalog}/support_demo/raw_data/
📖 Story: A payment system outage causes support ticket spike. Resolution times
degrade, enterprise customers churn, revenue drops $2.3M. With Databricks we
identify the root cause, affected customers, and prevent future impact.
| Table | Description | Rows | Key Assumptions |
|---|---|---|---|
| customers | Customer profiles with tier, MRR | 10,000 | Enterprise 10% but 60% of revenue |
| tickets | Support tickets with priority, resolution_time | 80,000 | Spike during outage, SLA breaches |
| incidents | System events (outages, deployments) | 50 | Payment outage mid-month |
| churn_events | Customer cancellations with reason | 500 | Spike after poor support experience |
Business metrics:
customers.mrr— Revenue at risk ($)tickets.resolution_hours— SLA performancechurn_events.lost_mrr— Churn impact ($)
The story this data tells:
- Incident table shows payment outage on March 15
- Tickets spike 5x during outage, resolution time degrades from 4h → 18h
- Enterprise customers with SLA breaches churn 3 weeks later
- Total impact: $2.3M lost MRR, traceable to one incident
- Databricks value: Root cause analysis, identify at-risk customers, build alerting
Ask user: "Does this story work? Any adjustments?"
Step 3: Ask About Data Features
- Skew (non-uniform distributions) - Enabled by default
- Joins (referential integrity) - Enabled by default
- Bad data injection (for data quality testing)
- Multi-language text
- Incremental mode (append instead of overwrite)
Pre-Generation Checklist
- Catalog confirmed - User explicitly approved which catalog to use
- Output location shown prominently in plan (easy to spot/change)
- Table specification shown and approved
- Assumptions about distributions confirmed
- User confirmed compute preference (Databricks Connect on serverless recommended)
- Data features selected
Do NOT proceed to code generation until user approves the plan, including the catalog.
Post-Generation Validation
Use databricks experimental aitools tools query to validate generated data (row counts, distributions, referential integrity). Query parquet files directly:
databricks experimental aitools tools query --warehouse $WAREHOUSE_ID "
SELECT COUNT(*) FROM parquet.\`/Volumes/CATALOG/SCHEMA/raw_data/customers\`
"
See references/2-troubleshooting.md for full validation examples.
Use Databricks Connect Spark + Faker Pattern
from databricks.connect import DatabricksSession
from pyspark.sql import functions as F
from pyspark.sql.types import StringType
import pandas as pd
# Setup serverless Spark session
spark = DatabricksSession.builder.serverless(True).getOrCreate()
# Pandas UDF pattern - import lib INSIDE the function (libs must be installed locally)
@F.pandas_udf(StringType())
def fake_name(ids: pd.Series) -> pd.Series:
from faker import Faker # Import inside UDF
fake = Faker()
return pd.Series([fake.name() for _ in range(len(ids))])
# Generate with spark.range, apply UDFs
customers_df = spark.range(0, 10000, numPartitions=16).select(
F.concat(F.lit("CUST-"), F.lpad(F.col("id").cast("string"), 5, "0")).alias("customer_id"),
fake_name(F.col("id")).alias("name"),
)
# Write to Volume as Parquet (default for raw data)
# Path is a folder with table name: /Volumes/catalog/schema/raw_data/customers/
spark.sql(f"CREATE SCHEMA IF NOT EXISTS {CATALOG}.{SCHEMA}")
spark.sql(f"CREATE VOLUME IF NOT EXISTS {CATALOG}.{SCHEMA}.raw_data")
customers_df.write.mode("overwrite").parquet(f"/Volumes/{CATALOG}/{SCHEMA}/raw_data/customers")
Partitions by scale: spark.range(N, numPartitions=P)
- <100K rows: 8 partitions
- 100K-500K: 16 partitions
- 500K-1M: 32 partitions
- 1M+: 64+ partitions
Output formats:
- Parquet to Volume (default):
df.write.parquet("/Volumes/.../raw_data/table")— raw data for pipelines - Delta Table:
df.write.saveAsTable("catalog.schema.table")— if user wants queryable tables - JSON/CSV: small dimension tables, replicate legacy systems
Performance Rules
Generated scripts must be highly performant. Never do these:
| Anti-Pattern | Why It's Slow | Do This Instead |
|---|---|---|
| Python loops on driver | Single-threaded, no parallelism | Use spark.range() + Spark operations |
.collect() then iterate | Brings all data to driver memory | Keep data in Spark, use DataFrame ops |
| Pandas → Spark → Pandas | Serialization overhead, defeats distribution | Stay in Spark, use pandas_udf only for UDFs |
| Read/write temp files | Unnecessary I/O | Chain DataFrame transformations |
| Scalar UDFs | Row-by-row processing | Use pandas_udf for batch processing |
Good pattern: spark.range() → Spark transforms → pandas_udf for Faker → write directly
Common Patterns
Weighted Categories (never uniform)
F.when(F.rand() < 0.6, "Free").when(F.rand() < 0.9, "Pro").otherwise("Enterprise")
Log-Normal Amounts (in a pandas UDF)
Use np.random.lognormal(mean, sigma) — always positive, long tail:
- Enterprise:
lognormal(7.5, 0.8)→ ~$1800 median - Pro:
lognormal(5.5, 0.7)→ ~$245 median - Free:
lognormal(4.0, 0.6)→ ~$55 median
Date Range (Last 6 Months)
END_DATE = datetime.now()
START_DATE = END_DATE - timedelta(days=180)
Infrastructure (always create in script)
spark.sql(f"CREATE SCHEMA IF NOT EXISTS {CATALOG}.{SCHEMA}")
spark.sql(f"CREATE VOLUME IF NOT EXISTS {CATALOG}.{SCHEMA}.raw_data")
Referential Integrity (FK pattern)
Write master table to Delta first, then read back for FK joins (no .cache() on serverless):
# 1. Write master table
customers_df.write.mode("overwrite").saveAsTable(f"{CATALOG}.{SCHEMA}.customers")
# 2. Read back for FK lookup
customer_lookup = spark.table(f"{CATALOG}.{SCHEMA}.customers").select("customer_idx", "customer_id")
# 3. Generate child table with valid FKs via join
orders_df = spark.range(N_ORDERS).select(
(F.abs(F.hash(F.col("id"))) % N_CUSTOMERS).alias("customer_idx")
)
orders_with_fk = orders_df.join(customer_lookup, on="customer_idx")
Setup
Requires Python 3.12 and databricks-connect>=16.4. Use uv:
uv pip install "databricks-connect>=16.4,<17.4" faker numpy pandas holidays
Related Skills
- databricks-unity-catalog — Managing catalogs, schemas, and volumes
- databricks-dabs — DABs for production deployment
Common Issues
| Issue | Solution |
|---|---|
ModuleNotFoundError: faker | Install locally: uv pip install faker, import inside UDF |
| Faker UDF is slow | Use pandas_udf for batch processing |
| Out of memory | Increase numPartitions in spark.range() |
| Referential integrity errors | Write master table to Delta first, read back for FK joins |
PERSIST TABLE is not supported on serverless | NEVER use .cache() or .persist() with serverless - write to Delta table first, then read back |
F.window vs Window confusion | Use from pyspark.sql.window import Window for row_number(), rank(), etc. F.window is for streaming only. |
| Broadcast variables not supported | NEVER use spark.sparkContext.broadcast() with serverless |
See references/2-troubleshooting.md for full troubleshooting guide.
Unity Catalog
Guidance for Unity Catalog governance — access control, the privilege model, external locations, securable DDL, and fine-grained access — plus system tables and volume file operations.
Before running
databricksCLI commands, confirm the CLI and the subcommand exist. Rundatabricks --version— this skill assumes the unified CLI (≥ v1.0.0). Several subcommands shown here (experimental aitools,system-schemas,external-lineage,grants) vary by version or workspace availability; if one is missing or rejects a flag, fall back to the SQL form or the Python SDK rather than guessing. Each reference notes its own version floor where relevant.
When to Use This Skill
Use this skill when:
Governance & access control (start here):
- Granting or revoking access —
GRANT/REVOKE, the UC privilege model, ownership (ALTER … OWNER TO),SHOW GRANTS, "who can read/write this table?" - Row- and column-level security — row filters, column masks, dynamic views with
current_user()/is_account_group_member() - External locations & storage credentials —
CREATE STORAGE CREDENTIAL,CREATE EXTERNAL LOCATION, backing external tables/volumes - Securable DDL & metadata — creating/altering catalogs, schemas, managed vs external tables, views; comments, tags, table properties, ownership
- Connection credentials & secrets: a connection's password from a secret scope (
secret()) and its rotation; where the secret lives
Observability & files:
- Working with volumes (upload, download, list files in
/Volumes/) - Querying lineage (table dependencies, column-level lineage)
- Analyzing audit logs (who accessed what, permission changes)
- Monitoring billing and usage (DBU consumption, cost analysis)
- Tracking compute resources (cluster usage, warehouse metrics)
- Reviewing job execution (run history, success rates, failures)
- Analyzing query performance (slow queries, warehouse utilization)
- Profiling data quality (data profiling, drift detection, metric tables)
Reference Files
| Topic | File | Description |
|---|---|---|
| Access Control | references/1-access-control.md | Privilege model, securable hierarchy, GRANT/REVOKE, ownership, inheritance, SHOW GRANTS |
| External Locations | references/2-external-locations.md | Storage credentials (AWS/Azure/GCP), external locations, validation |
| Securables DDL | references/3-securables-ddl.md | CREATE/ALTER/DROP catalogs/schemas/tables/views, comments, tags, ownership |
| Fine-Grained Access | references/4-fine-grained-access.md | Row filters, column masks, dynamic views |
| System Tables | references/5-system-tables.md | Lineage, audit, billing, compute, jobs, query history |
| Volumes | references/6-volumes.md | Volume file operations, permissions, best practices |
| Data Profiling | references/7-data-profiling.md | Data profiling, drift detection, profile metrics |
| Connections & Secrets | references/8-connections-and-secrets.md | A connection's password and its rotation; where the secret lives |
Quick Start
Create Unity Catalog Objects (CLI)
Use --json for create commands. Positional argument order differs per command and
has changed across CLI versions, so --json is the order-independent, version-stable form
shown throughout this skill.
# Create a catalog
databricks catalogs create --json '{"name": "my_catalog"}'
# Create a schema
databricks schemas create --json '{"name": "my_schema", "catalog_name": "my_catalog"}'
# Create a managed volume
databricks volumes create --json '{
"catalog_name": "my_catalog",
"schema_name": "my_schema",
"name": "my_volume",
"volume_type": "MANAGED"
}'
# List catalogs, schemas, volumes (read commands take simple positional args)
databricks catalogs list
databricks schemas list my_catalog
databricks volumes list my_catalog.my_schema
Positional create args still work if you prefer them, but the order is not uniform
across commands — this is the per-command order (and the reason --json is recommended):
| Command | Positional create order |
|---|---|
databricks catalogs create | NAME |
databricks schemas create | NAME CATALOG_NAME |
databricks volumes create | CATALOG_NAME SCHEMA_NAME NAME VOLUME_TYPE |
CLI surface varies by version. If a
databrickssubcommand or positional signature is missing in your install, prefer--json, the SQL form, or the Python SDK rather than guessing flags.
Volume File Operations (CLI)
databricks fs requires the dbfs: scheme prefix even for UC Volume paths — without it the CLI treats the path as local filesystem and errors with no such directory.
# List files in a volume
databricks fs ls dbfs:/Volumes/catalog/schema/volume/path/
# Upload a directory's contents to a volume (-r copies contents, not the directory itself)
databricks fs cp -r --overwrite /tmp/data dbfs:/Volumes/catalog/schema/volume/dest
# Download a file from a volume
databricks fs cp dbfs:/Volumes/catalog/schema/volume/file.csv /tmp/file.csv
# Create a directory in a volume
databricks fs mkdirs dbfs:/Volumes/catalog/schema/volume/new_folder
Grant & Revoke Access
GRANT/REVOKE is the core governance operation. See references/1-access-control.md for the full privilege model.
-- Grant read access on a schema to a group
GRANT USE CATALOG ON CATALOG analytics TO `data_readers`;
GRANT USE SCHEMA ON SCHEMA analytics.gold TO `data_readers`;
GRANT SELECT ON SCHEMA analytics.gold TO `data_readers`;
-- Who can access this table?
SHOW GRANTS ON TABLE analytics.gold.customers;
-- Revoke
REVOKE SELECT ON SCHEMA analytics.gold FROM `data_readers`;
Enable System Tables Access
-- Grant access to system tables
GRANT USE CATALOG ON CATALOG system TO `data_engineers`;
GRANT USE SCHEMA ON SCHEMA system.access TO `data_engineers`;
GRANT SELECT ON SCHEMA system.access TO `data_engineers`;
Common Queries
-- Table lineage: What tables feed into this table?
SELECT source_table_full_name, source_column_name
FROM system.access.table_lineage
WHERE target_table_full_name = 'catalog.schema.table'
AND event_date >= current_date() - 7;
-- Audit: Recent permission changes
SELECT event_time, user_identity.email, action_name, request_params
FROM system.access.audit
WHERE action_name LIKE '%GRANT%' OR action_name LIKE '%REVOKE%'
ORDER BY event_time DESC
LIMIT 100;
-- Billing: DBU usage by workspace
SELECT workspace_id, sku_name, SUM(usage_quantity) AS total_dbus
FROM system.billing.usage
WHERE usage_date >= current_date() - 30
GROUP BY workspace_id, sku_name;
Running SQL from the CLI
databricks experimental aitools tools queryis an experimental command. Theexperimentalnamespace is not guaranteed to be stable across CLI versions and may be absent in your install. Prefer running system-table SQL from a SQL warehouse (SQL editor, scheduled query) or the Python SDK (w.statement_execution.execute_statement), or a notebook. Use the experimental CLI only for quick ad-hoc checks.
Getting the IDs these examples use.
WAREHOUSE_ID— rundatabricks warehouses list(or copy it from a SQL warehouse's Connection details in the UI).METASTORE_ID(used in references/5-system-tables.md) —w.metastores.current().metastore_idvia the SDK, or the Catalog UI → metastore details.
Experimental CLI form (convenience only):
databricks experimental aitools tools query --warehouse WAREHOUSE_ID "
SELECT source_table_full_name, target_table_full_name
FROM system.access.table_lineage
WHERE event_date >= current_date() - 7
"
Stable SDK fallback (works on any CLI version):
from databricks.sdk import WorkspaceClient
w = WorkspaceClient()
resp = w.statement_execution.execute_statement(
warehouse_id="WAREHOUSE_ID",
statement="""
SELECT source_table_full_name, target_table_full_name
FROM system.access.table_lineage
WHERE event_date >= current_date() - 7
LIMIT 100
""",
)
for row in resp.result.data_array or []:
print(row)
CLI surface varies by version. If a
databrickssubcommand (e.g. anexperimentaltool,system-schemas, orexternal-lineage) is missing, fall back to the SQL warehouse or the Python SDK shown above rather than guessing flags.
Best Practices
- Grant minimal access - Apply least privilege; grant at the narrowest securable that works
- Filter by date - System tables can be large; always use date filters
- Use appropriate retention - Check your workspace's retention settings
- Schedule reports - Create scheduled queries for regular monitoring
- Prefer SQL/SDK over experimental CLI - For anything beyond quick checks
- Connection passwords - after a rotation, run
ALTER CONNECTIONwith every option and the samesecret(), ordatabricks connections updatefor a connection made withoutsecret(). Azure Key Vault-backed scopes: the vault's access-policy model only, and a vault in another tenant needs the creating user to be able to create service principals there. Where the secret lives: references/8-connections-and-secrets.md
Related Skills
This skill owns Unity Catalog governance: access control, the privilege model, external locations / storage credentials, securable DDL, fine-grained access, system tables, and volumes. For adjacent concerns, use the sibling skill instead:
- databricks-core (declared parent) — auth, profile selection, generic CLI, and catalog/table exploration
- databricks-metric-views — metric view definitions / DDL (
WITH METRICS LANGUAGE YAML) - databricks-iceberg — Managed Iceberg, External Iceberg Reads (fka Uniform), and Iceberg REST Catalog (IRC) credential vending for external engines — distinct from UC storage credentials (see references/2-external-locations.md)
- databricks-ml-training — UC model registration and
@prod/@challengeraliases - databricks-vector-search — Vector Search indexes
- databricks-pipelines, databricks-jobs, databricks-lakeflow-connect — producing tables via pipelines/jobs/managed ingestion
- databricks-lakebase — Lakebase / synced tables (OLTP)
- databricks-ai-functions — AI functions such as
ai_mask/ai_classify(AI transforms, not access control — see references/4-fine-grained-access.md) - databricks-aibi-dashboards — AI/BI dashboards on UC data
- databricks-synthetic-data-gen — generating data stored in UC volumes
Roadmap (not yet covered — deferred to a later version)
These governance areas are intentionally out of scope for v0.4.0 and planned for later:
- Delta Sharing / Marketplace / Clean Rooms
- Lakehouse Federation (connection setup + foreign catalogs; connection passwords: Best Practice 6)
- ABAC / governed tags as policy
Resources
Unity Gateway
Use the parent databricks-core skill for CLI authentication and profile selection.
Unity Gateway provides Unity Catalog securables for governed access to AI services. The CLI
command group keeps the earlier product name: databricks ai-gateway --help.
Choose the Correct Gateway
| Request | Use |
|---|---|
| Unity Gateway model service, MCP service, or model provider service | This skill and databricks ai-gateway |
AI Gateway configuration on a Model Serving endpoint, including put-ai-gateway | databricks-model-serving and databricks serving-endpoints |
| Lakeflow Connect ingestion gateway | databricks-lakeflow-connect |
Treat Unity AI Gateway as an earlier name for Unity Gateway when the resource is a Unity Catalog securable. Do not translate legacy Model Serving AI Gateway resources into Unity Gateway resources unless the user explicitly requests a migration.
CLI Workflow
Use the Databricks CLI for resource lifecycle and permission management. Do not substitute direct REST calls or a Databricks SDK for those operations. Python clients are supported for querying model and model provider services; read Query services.
An explicit request for Unity Gateway, Unity AI Gateway, or
databricks ai-gateway belongs to this skill. Do not switch to
databricks-model-serving merely because the request involves a model service or a
workspace. Use that skill only for a legacy per-endpoint AI Gateway operation or when a
provisioned-throughput destination requires Model Serving endpoint details.
-
Verify the CLI meets the minimum version:
databricks --version -
Inspect the command and the relevant operation before constructing a payload:
databricks ai-gateway --help databricks ai-gateway <operation> --help -
Pass create and update configuration through
--json. Use an explicit profile when profile-based authentication is required:databricks ai-gateway <operation> --json @payload.json --profile <PROFILE>
Do not invent JSON fields from similarly named legacy APIs. Build the payload from the installed CLI help and the Unity Gateway documentation for the selected service type.
Service Types
| Service type | CLI operations |
|---|---|
| Model service | create-model-service, get-model-service, list-model-services, update-model-service, delete-model-service; read Model services |
| MCP service | create-mcp-service, get-mcp-service, list-mcp-services, update-mcp-service, delete-mcp-service; read MCP services |
| Model provider service | create-model-provider-service, get-model-provider-service, list-model-provider-services, update-model-provider-service, delete-model-provider-service; read Model provider services |
Querying and permissions
- Before writing Python that invokes a model or model provider service, read references/querying.md.
- Before checking, granting, or revoking Unity Gateway access, read references/permissions.md.
- Before any model-service operation, read references/model-services.md. It contains the current resource-name conventions, JSON payload fields, required-input gate, and lifecycle commands.
- Before any MCP-service operation, read references/mcp-services.md for the equivalent MCP-specific contract.
- Before any model-provider-service operation, read references/model-provider-services.md.
- Before setting up a spend budget or spend alert on Unity Gateway usage, read references/budgets.md.
- Before any account-scoped operation (account groups as grant principals, account budgets), read references/account-access.md.
Do not fetch public documentation for fields already covered by these references. Consult the authoritative documentation only when a required field is absent from the reference or the user explicitly asks for the latest documentation. Do not assign a Beta or preview status unless the installed CLI help or current documentation explicitly does so.
Migration from Model Serving to Unity Gateway
When migrating from workspace-scoped Model Serving endpoints to Unity Catalog-scoped Unity Gateway services, read references/model-serving-migration.md.
Authoritative Documentation
- Unity Gateway overview
- Unity Gateway release notes
- Unity Gateway developer documentation
- Model service API reference
- MCP service API reference
- Model provider service API reference
- Query model services
- Open Responses specification
- Open Responses provider behavior on Databricks
- Query model provider services
- Model service requirements and grants
- Model provider service governance
- Routing and fallbacks
- Rate limits
- Register an external MCP server
Unstructured-Document for Demos and Eval Datasets on Databricks
Workflow for producing synthetic PDF documents + paired test questions as a Unity Catalog-resident dataset for Demos and RAG / unstructured-document retrieval evaluation on Databricks. The PDF-generation step uses standard local HTML → PDF tooling; the Databricks-specific value is the workflow shape — UC volume layout, paired question files, and integration with downstream Databricks retrieval / ai_extract / ai_parse_document evaluation.
Workflow
- Write HTML files to
./raw_data/html/(write multiple files in parallel for speed) — domain-shaped to match the documents your retrieval pipeline will see in production. - Convert HTML → PDF using
<SKILL_ROOT>/scripts/pdf_generator.py(parallel conversion, wrapsplutoprint). - Upload PDFs to a Unity Catalog volume via
databricks fs cp— same volume shape your production pipeline will read from. - Generate
./raw_data/pdf/pdf_eval_questions.jsonpairing each document with retrieval-eval questions; this becomes the gold dataset formlflow.genai.evaluate()or comparable retrieval-quality scorers.
If you only need ad-hoc PDFs (no Databricks workflow), any HTML → PDF tool (
weasyprint,wkhtmltopdf,playwright pdf,plutoprint) works directly — this skill exists for the synthetic-dataset-on-UC end-to-end shape, not as a general PDF generator.
Path convention:
<SKILL_ROOT>below = the directory containing this SKILL.md. Resolve to the absolute install path (e.g.~/.claude/skills/databricks-unstructured-pdf-generation)../raw_data/...paths are relative to your own project cwd.
Dependencies
uv pip install plutoprint
Step 1: Write HTML Files
mkdir -p ./raw_data/html
Write HTML documents to ./raw_data/html/filename.html. Use subdirectories to organize (structure is preserved).
Step 2: Convert to PDF
# Convert entire folder (parallel, 4 workers)
python <SKILL_ROOT>/scripts/pdf_generator.py convert --input ./raw_data/html --output ./raw_data/pdf
Skips files where PDF exists and is newer than HTML. Use --force to reconvert all.
Step 3: Upload to Volume
databricks fs requires the dbfs: scheme prefix even for UC Volume paths. -r copies the contents of the source directory into the target (the source directory name is not preserved), so name the target raw_data/pdf explicitly to keep the PDFs in their own folder on the volume. They land under raw_data/pdf/ — i.e. dbfs:/Volumes/my_catalog/my_schema/raw_data/pdf/report.pdf — so an ingest pipeline can point at that single folder. For a Genie Agent, upload to a dedicated managed volume instead (e.g. dbfs:/Volumes/my_catalog/my_schema/docs/), because Genie indexes the whole volume, not a subfolder (see databricks-genie-agents § Documents).
databricks fs cp -r --overwrite ./raw_data/pdf dbfs:/Volumes/my_catalog/my_schema/raw_data/pdf
Step 4: Generate Test Questions
Create ./raw_data/pdf/pdf_eval_questions.json with questions for evaluating a Genie Agent (or a legacy Knowledge Assistant) over the PDFs. Keep it out of a volume attached to a Genie Agent: the whole volume is indexed, so the answers would leak into the agent's context (upload the PDFs before writing it, or databricks fs rm it from that volume):
{
"api_errors_guide.pdf": {
"question": "What is the solution for error ERR-4521?",
"expected_fact": "Call /api/v2/auth/refresh with refresh_token before the 3600s TTL expires"
},
"installation_manual.pdf": {
"question": "What port does the service use by default?",
"expected_fact": "Port 8443 for HTTPS, configurable via CONFIG_PORT environment variable"
}
}
Use it to build Agent-mode test cases (or Genie benchmarks) and validate retrieval accuracy.
Document Content Guidelines
When generating documents for Genie Agent (document Q&A) testing or demos:
- Multi-page documents: Each PDF should be several pages with substantial content
- Specific error codes and solutions: Include product-specific error codes, causes, and resolution steps
- Technical details: API endpoints, configuration parameters, version numbers, specific commands
- Simple CSS: Keep styling minimal for fast HTML creation and reliable PDF conversion
- Queryable facts: Include details the agent must read the document to answer (not general knowledge)
Good document types:
- Product user manuals with troubleshooting sections
- API error reference guides (error codes, causes, solutions)
- Installation/configuration guides with specific steps
- Technical specifications with version-specific details
Example content: Instead of generic "Connection failed" errors, write:
- "Error ERR-4521: OAuth token expired. Cause: Token TTL exceeded 3600s default. Solution: Call
/api/v2/auth/refreshwith your refresh_token before expiration. See Section 4.2 for token lifecycle management."
CLI Reference
python <SKILL_ROOT>/scripts/pdf_generator.py convert [OPTIONS]
--input, -i Input HTML file or folder (required)
--output, -o Output folder for PDFs (required)
--force, -f Force reconvert (ignore timestamps)
--workers, -w Parallel workers (default: 4)
Folder Structure
Subfolder structure is preserved:
./raw_data/html/ ./raw_data/pdf/
├── report.html → ├── report.pdf
├── quarterly/ ├── quarterly/
│ └── q1.html → │ └── q1.pdf
└── legal/ └── legal/
└── terms.html → └── terms.pdf
Bundled Script
This skill ships one helper script:
| File | Description |
|---|---|
| scripts/pdf_generator.py | HTML → PDF converter (wraps plutoprint); parallel folder conversion with timestamp-skip. Referenced by Step 2 and the CLI Reference. |
The script ships at <SKILL_ROOT>/scripts/pdf_generator.py. If it is absent, recreate it from the CLI Reference above (a convert subcommand taking --input/--output/--force/--workers, wrapping plutoprint for HTML → PDF).
Troubleshooting
| Issue | Solution |
|---|---|
| "plutoprint not installed" | uv pip install plutoprint |
| PDF looks wrong | Check HTML/CSS syntax |
| "Volume does not exist" | databricks volumes create CATALOG SCHEMA VOLUME_NAME MANAGED (four separate positional args, not catalog.schema.volume) |
Databricks AI Search (formerly Vector Search)
FIRST: Use the parent databricks-core skill for CLI basics, authentication, and profile selection.
Patterns for creating, managing, and querying AI Search indexes for RAG and semantic search applications.
Naming: the REST API (/api/2.0/vector-search/...), the databricks vector-search-* CLI groups, and the databricks_vector_search_* Terraform resources keep the Vector Search names (API and tool names); the Python client is databricks-ai-search (AISearchClient).
When to Use
Use this skill when:
- Building RAG (Retrieval-Augmented Generation) applications
- Implementing semantic search or similarity matching
- Creating vector indexes from Delta tables
- Choosing between storage-optimized and standard endpoints
- Querying vector indexes with filters
Overview
Databricks AI Search provides managed vector similarity search with automatic embedding generation and Delta Lake integration.
| Component | Description |
|---|---|
| Endpoint | Compute resource hosting indexes (Standard or Storage-Optimized) |
| Index | Vector data structure for similarity search |
| Delta Sync | Auto-syncs with source Delta table |
| Direct Access | Manual CRUD operations on vectors (Standard endpoints only) |
Endpoint Types
| Type | Latency | Capacity | Cost | Best For |
|---|---|---|---|---|
| Standard | 20-50ms | 320M vectors (768 dim) | Higher | Real-time, low-latency, high QPS |
| Storage-Optimized | 300-500ms | 1B+ vectors (768 dim) | Up to 7x lower | 10M+ vectors, cost-sensitive |
Storage-Optimized supports only TRIGGERED Delta Sync indexes: no CONTINUOUS pipelines, no Direct Access, no target_qps. The endpoint type cannot be changed after creation.
Index Types
| Type | Embeddings | Sync | Use Case |
|---|---|---|---|
| Delta Sync (managed) | Databricks computes | Auto from Delta | Easiest setup |
| Delta Sync (self-managed) | You provide | Auto from Delta | Custom embeddings |
| Direct Access | You provide | Manual CRUD | Real-time updates |
Installation
pip install "databricks-ai-search>=0.79"
In a notebook, use %pip install instead and run dbutils.library.restartPython() afterwards:
the package requires mlflow-skinny>=3.10.1, so installing it upgrades mlflow in place.
databricks.ai_search first shipped in databricks-ai-search 0.73. This skill needs 0.79+:
from 0.79, get_endpoint, list_endpoints and list_indexes always return snake_case keys,
and 0.79 also includes the 0.78 fix that makes concurrent similarity_search calls thread-safe
(needed for production traffic). From 0.81, service principal auth no longer requires the
all-apis OAuth scope.
Existing databricks-vectorsearch code: on databricks-vectorsearch 0.74+,
VectorSearchClient is an alias of AISearchClient, so the calls in this skill work on it
once databricks-ai-search>=0.79 is installed alongside it. On 0.73 and older it is a
separate class: switch the import to from databricks.ai_search.client import AISearchClient.
The old class has no start/stop calls, and before 0.68 no target_qps.
Auth: a bare AISearchClient() uses the notebook's credentials. Elsewhere, pass
workspace_url plus a personal_access_token, or service_principal_client_id and
service_principal_client_secret (needed for high-QPS traffic; see
Production and High QPS).
AISearchClient takes plain strings (endpoint_type="STANDARD", pipeline_type="TRIGGERED").
Don't pass the databricks.sdk.service.vectorsearch enums to it; those are for
WorkspaceClient.vector_search_*.
Quick Start
Create Endpoint
from databricks.ai_search.client import AISearchClient
client = AISearchClient()
client.create_endpoint(
name="my-vs-endpoint",
endpoint_type="STANDARD" # or "STORAGE_OPTIMIZED"
)
# Note: Endpoint creation is asynchronous; check status with get_endpoint()
create_endpoint_and_wait(..., timeout=datetime.timedelta(minutes=30)) blocks until the endpoint is
ONLINE. It fails fast on OFFLINE (and, from 0.80, STOPPING and STOPPED); in other states it polls until the
timeout. On a timeout the endpoint may still be provisioning: check
client.get_endpoint(name)["endpoint_status"]["state"] before retrying, and don't create it again.
Create Delta Sync Index (Managed Embeddings)
Standard endpoints need Change Data Feed on the source table:
ALTER TABLE catalog.schema.documents SET TBLPROPERTIES (delta.enableChangeDataFeed = true);
# Source table must have: primary key column + text column
index = client.create_delta_sync_index(
endpoint_name="my-vs-endpoint",
source_table_name="catalog.schema.documents",
index_name="catalog.schema.my_index",
pipeline_type="TRIGGERED", # "CONTINUOUS" is Standard-only
primary_key="id",
embedding_source_column="content",
embedding_model_endpoint_name="databricks-gte-large-en"
)
An index on an endpoint with no other active indexes can wait for compute even while the endpoint says ONLINE. This typically takes 5-15 minutes, occasionally longer. Its status.detailed_state is PROVISIONING_ENDPOINT ("pending endpoint provisioning"), and sync() is rejected as not ready. The initial sync starts on its own. This also happens when an existing endpoint has released its compute after all indexes were deleted; deleting and recreating the only index can prolong the wait.
- From an agent's tool calls (which usually time out within minutes): check once per call with
databricks vector-search-indexes get-index <index> -o json. Finish whenstatus.detailed_statecontainsONLINEand does not containFAILED;readyalone is true even forONLINE_PIPELINE_FAILED. Stop onONLINE_PIPELINE_FAILEDorOFFLINE_FAILEDand reportstatus.messageandstatus.failed_status. Between checks, tell the user it is still provisioning. After 45 minutes total, report the current status and that provisioning has not finished; do not recreate the index or leave background sleep loops running. - From a notebook, job or long-running script: use the bounded 45-minute wait and failure diagnostics in Step 4 of the end-to-end walkthrough.
Query Index
index = client.get_index(
endpoint_name="my-vs-endpoint",
index_name="catalog.schema.my_index"
)
results = index.similarity_search(
query_text="What is machine learning?",
columns=["id", "content", "metadata"],
num_results=5
)
# Returns a dict; each row holds the requested columns, then the similarity score
for row in results["result"]["data_array"]:
print(row)
Common Patterns
Create Storage-Optimized Endpoint
# For large-scale, cost-effective deployments
client.create_endpoint(
name="my-storage-endpoint",
endpoint_type="STORAGE_OPTIMIZED"
)
Delta Sync with Self-Managed Embeddings
# Source table must have: primary key + embedding vector column
index = client.create_delta_sync_index(
endpoint_name="my-vs-endpoint",
source_table_name="catalog.schema.documents",
index_name="catalog.schema.my_index",
pipeline_type="TRIGGERED",
primary_key="id",
embedding_dimension=768,
embedding_vector_column="embedding"
)
Direct Access Index (Standard endpoints only)
# Create index for manual CRUD
index = client.create_direct_access_index(
endpoint_name="my-vs-endpoint",
index_name="catalog.schema.direct_index",
primary_key="id",
embedding_dimension=768,
embedding_vector_column="embedding",
schema={
"id": "string",
"text": "string",
"embedding": "array<float>",
"metadata": "string"
}
)
# Upsert data
index.upsert([
{"id": "1", "text": "Hello", "embedding": [0.1, 0.2, ...], "metadata": "doc1"},
{"id": "2", "text": "World", "embedding": [0.3, 0.4, ...], "metadata": "doc2"},
])
# Delete data
index.delete(primary_keys=["1", "2"])
Query with Embedding Vector
# When you have pre-computed query embedding
results = index.similarity_search(
query_vector=[0.1, 0.2, 0.3, ...], # Your 768-dim vector
columns=["id", "text"],
num_results=10
)
Hybrid Search (Semantic + Keyword)
Hybrid search runs ANN and full-text (keyword) search and merges the results with Reciprocal Rank Fusion (RRF). Use it when queries contain exact terms that must match — SKUs, error codes, proper nouns, or technical terminology — where pure semantic search might miss keyword-specific results. See references/search-modes.md for detailed guidance on choosing between ANN and hybrid search.
# Needs an index with a managed embedding model, so query_text alone is enough
index = client.get_index(
endpoint_name="my-vs-endpoint",
index_name="catalog.schema.my_index",
)
# Combines vector similarity with keyword matching
results = index.similarity_search(
query_text="SPARK-12345 executor memory error",
query_type="hybrid",
columns=["id", "content"],
num_results=10
)
Filtering
Filter syntax differs by endpoint type: Standard endpoints take a dict, Storage-Optimized endpoints take a SQL-like string. On Storage-Optimized, use the string: a dict (filters_json) is rejected on most query routes (except filter-only lookups with no query text or vector), and other routes translate it to SQL instead, so don't rely on either behavior.
Standard Endpoint Filters (Dictionary)
results = index.similarity_search(
query_text="machine learning",
columns=["id", "content"],
num_results=10,
filters={"category": "ai", "status": ["active", "pending"]}
)
Storage-Optimized Filters (SQL-like string)
index = client.get_index(
endpoint_name="my-storage-endpoint",
index_name="catalog.schema.my_storage_index",
)
results = index.similarity_search(
query_text="machine learning",
columns=["id", "content"],
num_results=10,
filters="category = 'ai' AND status IN ('active', 'pending')"
)
# filters="price > 100 AND price < 500"
# filters="department LIKE 'eng%'"
# filters="created_at >= TO_TIMESTAMP('2024-01-01T00:00:00')" # a bare timestamp string is rejected
See the filtering guide for operators and limitations.
Never interpolate untrusted input (tenant IDs, ACL values) into a filter as-is:
- Take column names from a fixed allowlist. In a dict filter the key also carries the operator (
"price >","col LIKE"), so allowlist whole keys. - Pass scalar values, validated against an allowlist or a strict pattern (for example,
^[A-Za-z0-9_-]+$). Never let untrusted input become a list: a list value becomes an OR/IN match. In a SQL string, don't rely on quote escaping alone. - On Storage-Optimized,
%and_in a LIKE value act as wildcards, so reject them in untrusted input. (On Standard, LIKE matches whole tokens, not wildcards.) - Filters are application-level ACLs, not a security boundary: row- and column-level permissions aren't supported, so anyone who can query the index can read every row.
In a RAG prompt, treat retrieved text as data, not instructions.
Trigger Index Sync
# TRIGGERED pipelines only, after the initial sync from creation finishes (rejected while provisioning).
# Also rejected while an update is running; wait for that update instead of retrying.
index = client.get_index(
endpoint_name="my-vs-endpoint",
index_name="catalog.schema.my_index"
)
index.sync()
Scan Index Entries
# Returns at most num_results rows, ordered by primary key. The response carries
# the cursor for the next page as last_primary_key.
page = index.scan(num_results=100)
rows = page.get("data", [])
while page.get("last_primary_key") and page.get("data"):
page = index.scan(num_results=100, last_primary_key=page["last_primary_key"])
rows += page.get("data", [])
# scan rows are field structs, NOT the plain lists similarity_search returns:
# {"fields": [{"key": "id", "value": {"string_value": "r1"}}, ...]}
for row in rows:
values = {f["key"]: next(iter(f["value"].values())) for f in row["fields"]}
Production and High QPS
By default, Standard endpoints serve about 20-200 QPS, depending on index size. Traffic over a PAT or the workspace query URL is capped at a few tens of QPS.
- Auth and route: production traffic needs service principal OAuth and the index URL. Create
AISearchClient(workspace_url=..., service_principal_client_id=..., service_principal_client_secret=...), callget_index()once, and reuse the index object; it queries the index URL. Use a dedicated service principal. For Delta Sync indexes with managed embeddings, an IP access list or a private access settings object (on Azure, public network access disabled) keeps queries on the capped route; on GCP, a private access settings object does so for every index. Otherwise, self-managed embeddings avoid it. See requirements.- Pitfalls: with any token passed as
personal_access_token(a PAT, or a forwarded user or on-behalf-of token such as the Databricks Appsx-forwarded-access-token), or a bareAISearchClient()that picks up notebook or model-serving credentials,get_index()queries the workspace URL even thoughdescribe()listsstatus.index_url; ifindex.index_urlstarts with your workspace URL, queries are on the capped route. In Databricks Apps, send query traffic with the app's service principal, not the forwarded user token.workspace_urlmust includehttps://: in Databricks Apps,DATABRICKS_HOSTcan lack the scheme, so add it (example).
- Pitfalls: with any token passed as
- Query type: start with ANN; hybrid uses about 4x the resources. Load-test with the query type production sends.
- 429s: move to service principal OAuth and the index URL; wait for
SCALING_CHANGE_APPLIED; withquery_vector, lower concurrency or raisetarget_qps; withquery_texton a managed-embedding index, the embedding endpoint may be the bottleneck.similarity_searchalready retries (total_retries,backoff_factor,backoff_jitter); raise those or add your own bounded backoff with jitter. See Troubleshoot 429 errors. - Load testing: use Run load test on the endpoint page or the load test notebook. With
databricks-ai-search0.82+,databricks-ai-search guide --topic high-qpsprints the guide anddatabricks-ai-search load-test --index <name>plans a run (it runs only with--yes); skip this if thedatabricks-ai-searchcommand isn't available. - Agent safety: ask before creating endpoints or indexes, setting
target_qps, running load tests, or upserting/deleting data, since each costs money or changes data. Never print, paste, or commit tokens or client secrets; read them from a secret scope or environment variables.
Target QPS is Standard-only and best-effort, and the capacity is billed whether or not traffic arrives. Storage-Optimized endpoints reject target_qps. A change can be rejected with RESOURCE_CONFLICT while a previous change is still applying. Nothing is saved, so requested_target_qps still shows the earlier target: poll until scaling_info.state is SCALING_CHANGE_APPLIED (whatever the target), or until scaling_info.target_qps_change_allowed_after_timestamp if set (epoch ms; int() then /1000), otherwise the minutes the error gives, then call update_endpoint again before running the loop below.
import time
from datetime import datetime, timezone
client.update_endpoint(name="my-vs-endpoint", target_qps=500)
# Or at creation: create_endpoint(..., target_qps=500).
# Create the index before this loop: target QPS may not apply until the endpoint has an index.
# Match the requested target too: right after the update, state can still be APPLIED from an earlier target.
# Target QPS is best-effort: if it's still in progress at the deadline, move on.
deadline = time.monotonic() + 30 * 60
while time.monotonic() < deadline:
info = client.get_endpoint("my-vs-endpoint").get("scaling_info") or {}
# int(): proto3 JSON may serialize int64 fields as strings
if info.get("state") == "SCALING_CHANGE_APPLIED" and int(info.get("requested_target_qps") or 0) == 500:
break
progress = {k: v for k, v in info.items() if k in ("state", "phase", "index_initialization_percentage", "delayed_reason")}
if info.get("estimated_completion_timestamp"):
eta = datetime.fromtimestamp(int(info["estimated_completion_timestamp"]) / 1000, tz=timezone.utc)
progress["eta"] = eta.isoformat()
print(progress)
time.sleep(30)
print(info)
While state is SCALING_CHANGE_IN_PROGRESS, scaling_info also reports progress (Beta; every field is optional and can hold other values, so check each one and report unknown values as-is):
estimated_completion_timestamp: expected completion, epoch ms (int()then /1000). It can change as scaling progresses, and it's unset when no estimate is available yet or whendelayed_reasonis set.phase:ESTIMATING,ADJUSTING_CAPACITY, orINITIALIZING_INDEX(withindex_initialization_percentage, 0-100).delayed_reason:CAPACITY_UNAVAILABLE,SCALE_DOWN_BLOCKED, orTAKING_LONGER_THAN_EXPECTED.
Report the ETA, phase, or delay reason to the user instead of guessing, and keep polling until SCALING_CHANGE_APPLIED or the loop's deadline. If all three are unset, that isn't a hang (some endpoints don't report progress): check the endpoint has an index and keep polling state.
ONLINE doesn't mean scaled. These are separate waits:
- Endpoint provisioned:
endpoint_status.stateisONLINE(whatcreate_endpoint_and_waitwaits for).ONLINEdoesn't mean the target QPS is applied. - Target QPS applied:
scaling_info.stateisSCALING_CHANGE_APPLIEDfor the requested target (the loop above, run after the index is created). This takes minutes to hours, depending on index size and how much the target changes. - Index ready: wait as in Create Delta Sync Index. After
PROVISIONING_ENDPOINT, a Delta Sync index typically moves throughPROVISIONING_INDEX,PROVISIONING_PIPELINE_RESOURCES, andPROVISIONING_INITIAL_SNAPSHOT; a Direct Access index goes fromPROVISIONING_INDEXtoONLINE_DIRECT_ACCESS. A Direct Access index can stay inPROVISIONING_INDEXwith a message that creation has failed; stop and report it.
Don't create a second endpoint to get around either wait: it repeats endpoint and index provisioning, and without target_qps it lacks the target capacity.
Load-test only after all three; a load test before SCALING_CHANGE_APPLIED doesn't reflect the target capacity. If the loop above hits its deadline, report that target QPS is still applying instead of load testing.
Search Quality Evaluation (Beta)
Retrieval quality evaluation generates queries from a managed Delta Sync index, runs them with ANN, hybrid and full-text (each with and without the reranker), scores results with an LLM judge, and reports DCG@10 and other metrics.
- Start it from the index page in Catalog Explorer. In the Search Quality Evaluation section, click Evaluate search quality (or Start new evaluation / Create evaluation under Latest evaluation runs after a run). The built-in Beta evaluation is not configurable. There is no CLI command, SDK method or documented REST API for this built-in evaluation, so don't probe for one. If you can't use the browser, finish the index, then give the user the index page link and these steps, and pick up again from the results.
- Requirements: a workspace admin turns on AI Search: Quality Evaluation from the workspace Previews page; the person starting a run needs SELECT on the index and may also need CREATE TABLE on its schema, because evaluation writes output tables there. Each run is a job that costs money; its status shows on the index page.
- Reading results: the index page shows the latest runs and their DCG@10; View results opens the dashboard, and What do my results mean? asks Genie Code to interpret the run. Judged metrics only see the text you indexed: if users search on attributes that live in other columns or tables, also check results against those structured fields, since the judge can't see what's missing.
Reference Files
| Topic | File | Description |
|---|---|---|
| Index Types | references/index-types.md | Detailed comparison of Delta Sync (managed/self-managed) vs Direct Access |
| End-to-End RAG | references/end-to-end-rag.md | Complete walkthrough: source table → endpoint → index → query → agent integration |
| Search Modes | references/search-modes.md | When to use semantic (ANN) vs hybrid search, decision guide |
| Operations | references/troubleshooting-and-operations.md | Monitoring, cost optimization, capacity planning, migration |
CLI Quick Reference
# List endpoints
databricks vector-search-endpoints list-endpoints
# Create endpoint (positional args: NAME ENDPOINT_TYPE)
databricks vector-search-endpoints create-endpoint my-endpoint STANDARD
# List indexes on endpoint (positional arg: ENDPOINT_NAME)
databricks vector-search-indexes list-indexes my-endpoint
# Get index status (positional arg: INDEX_NAME)
databricks vector-search-indexes get-index catalog.schema.my_index
# Sync index (positional arg: INDEX_NAME)
databricks vector-search-indexes sync-index catalog.schema.my_index
# Delete index (positional arg: INDEX_NAME)
databricks vector-search-indexes delete-index catalog.schema.my_index
Common Issues
| Issue | Solution |
|---|---|
| Index sync slow | Use Storage-Optimized endpoints (10-20x faster indexing) |
| Query latency high | Use Standard endpoint for <100ms latency |
| Filters not working | Standard endpoints use a dict (filters={"col": "val"}); Storage-Optimized use a SQL string (filters="col = 'val'") |
| Embedding dimension mismatch | Ensure query and index dimensions match |
| Index not updating | Check pipeline_type; use index.sync() for TRIGGERED. If it's rejected as already running, wait for the current update to finish |
Index "pending endpoint provisioning" / sync() "not ready" | Normal for an index on an endpoint with no other active indexes, even while the endpoint says ONLINE; wait (see Create Delta Sync Index) instead of recreating it |
| Evaluating search quality | Start it from the index page; there's no CLI or SDK command. See Search Quality Evaluation |
| Out of capacity | Capacity is sized from the source table at index creation. Load the full table first; past ~320M vectors, recreate on Storage-Optimized (1B+ vectors) |
| Rate limited (429 errors) | See Production and High QPS |
query_vector truncated | Large vectors (e.g. 1024-dim) can be truncated when serialized as JSON. Use query_text instead (for managed embedding indexes), or pass the vector directly via index.similarity_search(query_vector=...) |
Embedding Models
Databricks provides built-in embedding models:
| Model | Dimensions | Context Window | Use Case |
|---|---|---|---|
databricks-qwen3-embedding-0-6b | 1024 (configurable, up to 1024) | ~32K tokens | Recommended for production by the AI Search docs (Public Preview) |
databricks-gte-large-en | 1024 | 8192 tokens | English text, high quality |
databricks-bge-large-en | 1024 | 512 tokens | English text, general purpose |
# Use with managed embeddings
index = client.create_delta_sync_index(
...,
embedding_source_column="content",
embedding_model_endpoint_name="databricks-qwen3-embedding-0-6b"
)
Notes
- Storage-Optimized is newer — better for most use cases unless you need <100ms latency, high QPS, Direct Access, or CONTINUOUS sync
- Delta Sync recommended — easier than Direct Access for most scenarios
- Hybrid search — available for both Delta Sync and Direct Access indexes
columns_to_sync— limits which columns are synced and returned; only synced columns are available in query results. On Storage-Optimized, the docs' limitations list says it's unsupported, but their full-text example uses it; prefer syncing all columns there- Filter syntax differs by endpoint — Standard uses a dict, Storage-Optimized uses a SQL-like string
- Management vs runtime — CLI and SDK handle lifecycle management; for agent tool-calling at runtime, use
VectorSearchRetrieverToolfromdatabricks_langchainordatabricks_openai
Related Skills
- databricks-model-serving - Deploy agents that use VectorSearchRetrieverTool
- databricks-ml-training - Author and deploy a
ResponsesAgentthat uses an AI Search retriever - databricks-genie-agents - No-code document Q&A: a Genie Agent with a UC volume (replaces the legacy Knowledge Assistant); hand-roll AI Search only for custom retrieval
- databricks-unstructured-pdf-generation - Generate documents to index in AI Search
- databricks-unity-catalog - Manage the catalogs and tables that back Delta Sync indexes
- databricks-pipelines - Build Delta tables used as AI Search sources
Zerobus Ingest
Zerobus Ingest (part of Lakeflow Connect) is a managed, push-based ingestion service: a producer sends records and they land in a Unity Catalog managed Delta table, with no broker or ingestion cluster to run. The SDKs do real work for you: OAuth, buffering, offset tracking, durability acknowledgments, and automatic recovery that replays unacknowledged records after transient failures.
FIRST: Use the parent databricks-core skill for CLI install, authentication, and profile selection.
Decide before you build
Recommend Zerobus when data is pushed from an app, service, or device; the destination is a Delta table; the workload is continuous or event-driven and needs a shock absorber (Zerobus lands data in about 5 seconds, and it still fits when minutes are acceptable); and there is one main downstream consumer (or a message bus such as Apache Kafka, Amazon Kinesis, Azure Event Hubs, or Google Cloud Pub/Sub is just a pipe to the lakehouse).
Don't recommend Zerobus when many independent consumers outside the lakehouse need the data (use a message bus such as Apache Kafka, Amazon Kinesis, Azure Event Hubs, or Google Cloud Pub/Sub), or when the data is already in Kafka or another bus and must be pulled into the lakehouse (use the Lakeflow Connect managed Kafka connector or Spark Structured Streaming). Explain why and name the alternative. Before ruling Zerobus out for files, SaaS apps, or databases, check references/choose-an-interface.md: batches of columnar data can go through Arrow Flight, and a SaaS app that can push to a Zerobus-compatible API can use Zerobus.
Prefer an off-the-shelf path. If the source already speaks OpenTelemetry, Kafka, or MQTT, or runs Debezium, Vector, Telegraf, or Cribl, point the user to that path instead of writing SDK code. All of these, plus the full decision tree, are in references/choose-an-interface.md. For existing Kafka producers, read references/kafka.md; for MQTT v5 devices, references/mqtt.md; for columnar batches (Parquet, Arrow, DataFrames), references/arrow-flight.md; for migrating a pipeline from Kafka or a Kafka-compatible system, read references/migration-from-kafka-compatible-systems.md.
Workflow
Treat the request as a conversation: ask one question at a time, use a multiple-choice question tool if one is available, and always offer "Not sure, help me decide". Skip any question the user has already answered.
Many users won't know Databricks. If the user is an app, DevOps, or platform engineer who's new to Databricks, explain each Databricks term in plain language the first time you use it, map it to something they know (a table name is like database.schema.table in Postgres; a service principal is like a service account), and offer to set up what's missing for them. Read references/new-to-databricks.md.
Step 0: Show your plan
List the steps you'll run and what you'll create (table, grants, producer file). Ask before creating any catalog, schema, table, or service principal; never assume the user wants a new one.
Step 1: Ask where the producer runs
Where will the producer that sends data to Zerobus run?
- Outside Databricks: a service, app, device or gateway, or my laptop
- Inside Databricks: a Lakeflow Job or a Databricks App
- Not sure, help me decide
- Outside Databricks (most common): write the producer as a local file the user keeps; read credentials from environment variables.
- Inside Databricks: read references/databricks-apps.md. The Python SDK runs on classic and serverless compute; install it as a job, cluster, or environment dependency.
- Not sure: ask what generates the data. Data almost always originates outside Databricks, so default there.
Step 2: Ask only the questions you still need
Questions come in three groups. Ask the required ones first: they're all you need to build a first client. Ask the advanced ones only when the user raises the topic, or when the answer would change your recommendation or the deployment.
Required to build a client:
- What is the data, and where does it come from? Application events, logs, metrics, traces, CDC from a database, IoT or sensor readings, clickstream? This decides the route: traces, logs, and metrics often fit OTLP, database changes fit Debezium, and an existing Kafka producer fits the Kafka-compatible APIs (references/choose-an-interface.md).
- Target table: the three-part name, whether it exists yet, and what shape the records have (a sample event helps). Must any fields always be present, and will the shape change over time (references/schema-management.md)?
- Language and runtime: and can the environment use precompiled binaries / native libraries? The Python, TypeScript, Java, Go (cgo), C++, and .NET SDKs load a prebuilt Rust core. If the user doesn't know, assume precompiled binaries are allowed, and change course only if the SDK fails to install or load. If the platform has no prebuilt binary (Alpine, glibc older than 2.34 for Python), or the user wants to build the SDK themselves, build it from source following that SDK README's build steps (references/build-from-source.md). If precompiled binaries aren't allowed, use a pure SDK: Rust or pure Go (pure C doesn't send data yet). Edge runtimes such as Cloudflare Workers or Deno Deploy can't load native code, so use REST there (references/rest.md).
If the user doesn't name a language, default to Python.
Advanced: is Zerobus the right service? Ask when the fit is in doubt, for example when other systems also read the data or the user mentions strict volume or freshness needs (references/choose-an-interface.md):
- Who consumes it? Is the lakehouse the one main destination, or do several independent systems read the same data? Many consumers outside the lakehouse, replay, or consumer groups mean keep the message bus (don't recommend Zerobus as its replacement).
- Volume and latency: rough records per second and how fresh the data must be. Zerobus makes records durable in about 150 ms and queryable in about 5 seconds, and it still fits when minutes are acceptable. Volume decides how many streams to open later (references/performance.md); it doesn't change how you build the first client.
Advanced setup: what it takes to run in the user's environment. Ask before deploying beyond a laptop, or when a connection or auth step fails:
- Network: public internet, front-end PrivateLink, an HTTP proxy, or on-prem (references/networking.md).
- Identity: does the organization require identity federation (for example Entra ID) instead of a service principal secret (references/authentication.md)?
Step 3: Confirm prerequisites
Details are in references/authentication.md. Don't run a producer until every row passes. If something is missing, offer to create it: design the table from a sample event, create the service principal, and grant it, one approved step at a time (references/new-to-databricks.md).
| # | Prerequisite | Verify |
|---|---|---|
| 1 | Workspace in a supported region | Region is listed under availability on the quotas page |
| 2 | Zerobus endpoint, e.g. https://<workspace-id>.zerobus.<region>.cloud.databricks.com (AWS) | Uses the numeric workspace ID and the workspace's region |
| 3 | A pre-created managed Delta table. Zerobus never creates or alters tables | databricks experimental aitools tools discover-schema <catalog.schema.table> --profile <PROFILE> shows the expected columns |
| 4 | A service principal with an OAuth client ID and secret | databricks service-principals list --profile <PROFILE> lists it |
| 5 | Grants: USE CATALOG, USE SCHEMA, and MODIFY + SELECT directly on the table. ALL PRIVILEGES is not sufficient; grant these four explicitly | SHOW GRANTS ON TABLE <catalog.schema.table> lists the service principal with MODIFY and SELECT |
| 6 | Outbound HTTPS (443) from the producer to the endpoint and the workspace | Connectivity check in references/networking.md |
Step 4: Build the producer
Read the SDK reference for the chosen language. For production, prefer Protobuf over JSON (references/protobuf-schema.md). Never hard-code secrets.
Always write a non-blocking producer. Use this ladder, and only go further down when the user needs it:
| Need | Use | Blocks? |
|---|---|---|
| Send records (the default) | ingest_record_offset() / ingest_records_offset(): queues and returns; the SDK sends and tracks acks in the background | No |
| Act while ingestion runs: metrics, logging, progress, errors, checkpoints | An ack callback (on_ack / on_error) | No |
| Durability boundary: end of a bounded batch, before shutdown | flush() once, then close() | Yes, once |
| Advanced: must not continue until one specific record is durable | wait_for_offset(offset) | Yes. Never per record or in a loop |
Ack callbacks exist in Python, Java, Rust, C++, and pure Go. The TypeScript, CGO Go, and .NET SDKs don't document one; there, call flush() on an interval for progress. Method names differ by language; use the SDK reference.
Minimal Python producer (APIs from the SDK README), with an ack callback for progress and errors:
import os
from zerobus.sdk.sync import ZerobusSdk
from zerobus.sdk.shared import AckCallback, StreamConfigurationOptions, TableProperties
class Progress(AckCallback):
def on_ack(self, offset: int): # runs on an SDK background thread; keep it fast
print(f"durable through offset {offset}")
def on_error(self, offset: int, error_message: str):
print(f"error at offset {offset}: {error_message}")
sdk = ZerobusSdk(os.environ["ZEROBUS_SERVER_ENDPOINT"], os.environ["DATABRICKS_WORKSPACE_URL"])
stream = sdk.create_stream(
os.environ["DATABRICKS_CLIENT_ID"],
os.environ["DATABRICKS_CLIENT_SECRET"],
TableProperties(os.environ["ZEROBUS_TABLE_NAME"]),
StreamConfigurationOptions(ack_callback=Progress()),
)
try:
for i in range(100):
stream.ingest_record_offset({"device_name": f"sensor-{i % 10}", "temp": 20 + (i % 15)}) # queues, returns
stream.flush() # durability boundary: blocks once until everything queued is acknowledged
finally:
stream.close()
Step 5: Send a small test batch
Send 10 to 100 records first. A durability acknowledgment (an on_ack callback, or flush() returning) means Zerobus stored the records safely. It does not mean they are queryable yet.
Step 6: Verify the rows landed
Wait about 5 seconds after the acknowledgment (rows become queryable about 5 seconds after they're sent), then run:
databricks experimental aitools tools query \
"SELECT count(*) AS row_count, max(<timestamp_column>) AS latest FROM <catalog.schema.table>" \
--profile <PROFILE>
Expected: row_count went up by the number of records you sent, and latest is close to the send time. If not, read references/troubleshooting.md.
Step 7: Harden for production
Before the user scales up, cover schema evolution, duplicates, throughput, networking, and monitoring using the reference map below.
Key facts
- Table first. Update the table schema before you update producers. Fields that don't fit can go to a rescue column (Beta, JSON only).
- Durable is not queryable. Zerobus materializes acknowledged data into the Delta table as a separate step: durable in about 150 ms, queryable in about 5 seconds (how-it-works.md).
- At-least-once delivery. Recovery or a client retry can resend records, so a few duplicates are possible. Most workloads accept them; don't add deduplication by default, and never claim exactly-once. If the user needs unique rows, see how-it-works.md.
- Ordering is per stream only. No global ordering across streams (how-it-works.md).
- Non-blocking by default (the Step 4 ladder). Ingest calls block only on backpressure: records wait in an in-memory in-flight buffer until acked, and calls block only when it's full (references/how-it-works.md).
- Recovery is built in. The SDK reconnects and replays unacked records. Don't hand-roll a retry loop; custom recovery is advanced (references/recovery.md).
- TIMESTAMP columns take epoch microseconds (an integer), not strings.
- Quotas are adjustable defaults, set per stream and per table. Add streams before asking for an increase (performance.md).
Common issues
Match the symptom in references/troubleshooting.md (connection, auth and grants, schema, throughput, visibility, duplicates, install / native binaries). Most common: error 4024 = grants not directly on the table; REST 401 after about an hour = expired token.
SDKs: record formats and working examples
Record formats per SDK (JSON, Protobuf, Avro, Arrow Flight) and each SDK's working examples folder: choose-an-interface.md § SDKs. Open its examples before writing code for a specific API. Don't attach a release status to an SDK; use that table. Pure C doesn't send data yet.
Reference map: read only what the task needs
These references hold the decisions, gotchas, and tested minimal samples for each route. For the full API surface, other record types, and the latest SDK versions, fetch the SDK README linked in each file's Sources (raw Markdown); for working code, use the examples folders listed in choose-an-interface.md. docs.databricks.com pages are HTML. Long runnable programs for MQTT and the raw gRPC API are in examples/.
| Read | When |
|---|---|
| new-to-databricks.md | The user doesn't know Databricks: plain-language glossary, and a guided first-time setup (pick a catalog and schema, design the table from a sample event, create the service principal, grant, save settings) |
| choose-an-interface.md | Unsure whether to use Zerobus, the SDK, REST, or a protocol/tool path; checking runtime compatibility |
| how-it-works.md | How Zerobus works: streams, the data path (the async ack loop, then batch commits to Delta), durable vs queryable, how the SDK processes records (in-flight buffer, cumulative acks, backpressure), ordering, delivery and duplicates, scaling |
| recovery.md | Stream failures, retryable vs fatal errors, advanced custom recovery (get_unacked_records, recreate_stream), durable-fallback reprocessing |
| authentication.md | Endpoints per cloud, service principal OAuth (official), Entra ID / identity federation (SDK-supported, docs pending), grants, env vars, local vs deployed credentials, table-scoped tokens for REST, Kafka, MQTT, and gRPC (mint and refresh), auth errors |
| python-sdk.md | Python producers: install and platforms, a Protobuf producer, ack callbacks; links to the README for async and other APIs |
| typescript-sdk.md | Node.js producers and native-binary requirements |
| go-sdk.md | Go producers with cgo (go/) or pure Go (purego/) |
| java-sdk.md | JVM producers |
| other-sdks.md | Rust, C++, C#/.NET |
| kafka.md | Existing Kafka producers: Apache Kafka-compatible producer APIs (Beta), connection settings, token callback, limits, errors |
| mqtt.md | MQTT v5 devices and gateways: connection settings, CONNECT auth, limits, errors; a runnable non-blocking QoS 1 publisher with token refresh on reconnect in examples/mqtt_producer.py |
| arrow-flight.md | Columnar batches (Parquet files, Arrow, pandas or Polars DataFrames): schema matching, a non-blocking Python producer, links for other languages, tuning, recovery |
| migration-from-kafka-compatible-systems.md | Migrating a pipeline from Kafka or a Kafka-compatible system into Delta: when to migrate, the four pathways, step-by-step cutover, rollback |
| build-from-source.md | The user wants to build the SDK themselves, or there's no prebuilt binary (Alpine/musl, older glibc, other CPUs): which platforms each SDK ships, each SDK README's source-build steps (Python wheel via make build, TypeScript, Java with JNI, Go, .NET, C++, Rust), an Alpine Dockerfile, build errors |
| grpc-api.md | Building a client from scratch against the gRPC API: the .proto, stub generation, the EphemeralStream messages, auth headers, the async ack loop, recovery; a runnable Python client in examples/grpc_client.py |
| rest.md | Edge runtimes and serverless functions outside Databricks (Cloudflare Workers, Deno Deploy), languages without an SDK, low-volume senders. Uses a table-scoped token from authentication.md (tokens expire after about an hour) |
| protobuf-schema.md | Generating .proto from a table, type mappings |
| schema-management.md | The table as the contract: how records are matched, the three strictness scenarios, the rescue column (setup, tag, routing rules), safe schema evolution, Protobuf rules |
| performance.md | Batching, stream count, quotas, table layout |
| networking.md | Ports and TLS per interface, front-end PrivateLink, on-prem producers and HTTP proxies, firewalled storage, cross-region, connectivity checks |
| observability.md | The monitoring dashboard bundle (install), ack callbacks, system tables, latency (ack and end-to-end freshness), errors, alerts |
| troubleshooting.md | Any error or missing data |
| databricks-apps.md | Running the producer in a Databricks App or Lakeflow Job |
Related skills
- databricks-core: CLI, profiles, auth, and ad hoc SQL for verification
- databricks-unity-catalog: catalogs, schemas, tables, and grants
- databricks-pipelines: downstream processing of ingested data, including a deduplicated table built with AUTO CDC
- databricks-lakeflow-connect-saas: managed connectors when Databricks should pull from a SaaS app, Kafka or RabbitMQ
- databricks-lakeflow-connect: managed connectors when Databricks should pull from a database
- databricks-apps / databricks-apps-python: building the Databricks App that hosts a producer
- databricks-dabs: deploying a producer job with Declarative Automation Bundles
- databricks-aibi-dashboards: building a dashboard over the ingested events or telemetry
Commands 2
Databricks Doctor
Run a read-only health check and report a short status table. Make no
changes; every step below only reads. If a subcommand or flag is unfamiliar,
check databricks <group> --help first rather than guessing.
Run these in order. Don't stop on the first failure; collect what you can and report the rest as unknown.
- CLI:
databricks --version. Flag only if it's missing; don't gate on a specific version (the CLI surfaces its own update notice). - Profiles:
databricks auth profiles. List configured profiles and validity. If$1is given, use that profile for the rest. Otherwise, if more than one profile exists, ask the user which to use (never auto-select). - Auth method:
databricks auth describe --profile <profile>shows the effective host, user, and credential source (never pass--sensitive). - Auth validity:
databricks current-user me --profile <profile>. This single API call proves the credentials work end to end (token valid, workspace reachable, expected identity); don't probe other APIs for it. For account-level profiles (anaccounts.*host),current-user medoes not exist; report whatauth describeresolved instead.
Stop here by default. Run the extended checks below only when the user passed full or asked about compute or jobs:
- Compute:
databricks warehouses listanddatabricks clusters listfor the profile. Note what's running. - Recent job failures: list recent job runs (e.g.
databricks jobs list-runs --limit 20 --profile <profile>) and surface any recent failures.
Then print a compact table: check | status (✅/⚠️/❌) | detail. End with the
single most useful next action (e.g. "run /databricks:setup to add a profile").
This is a status check; it only reads, so don't run anything that changes state.
Databricks Setup
Guide the user through Databricks CLI authentication. Use the databricks-core skill for the authoritative auth details; this command is the step-by-step wrapper around it.
- CLI present?
databricks --version. If it's missing, follow the install steps in the databricks-core skill (databricks-cli-install.md). In sandboxed environments (Cursor, containers), print the install command and ask the user to run it in their own terminal. Don't try to install into the sandbox. - Existing profiles?
databricks auth profiles. Show what's already configured. If a working profile exists, ask whether to reuse it or add a new one. If no profiles exist and the user has no Databricks account yet, don't search the web for signup options. Point them to the free trial at https://www.databricks.com/signup/express?provider=DB&dbx_source=agent-skills&dbxSource=agent-skills&utm_source=agent-skills&utm_campaign=setup (startups can also apply for credits at https://www.databricks.com/product/startups), wait for them to paste the new workspace URL, then continue at step 3 with that URL. - Pick an auth method (ask the user;
$1may be a workspace or account console URL):- OAuth U2M (default, interactive):
databricks auth login --host <workspace-url> --profile <name>. Opens a browser. Best for laptops. If the user doesn't know their workspace URL, plaindatabricks auth login --profile <name>opens login.databricks.com to sign in and pick a workspace. URLs copied from the browser may carry?w=<workspace-id>oraccount_id=query params; the CLI accepts them, but quote the URL so the shell doesn't interpret the?. - Account-level: when the host is an account console URL
(
accounts.cloud.databricks.com,accounts.azuredatabricks.net,accounts.gcp.databricks.com), also pass the account ID:databricks auth login --host <account-url> --account-id <uuid> --profile <name>. Ask for the account ID if it isn't in the URL (it's the UUID shown in the account console address bar). - PAT:
databricks configure --token --profile <name>; the user pastes a personal access token. This command prompts on stdin, so don't run it yourself (it hangs without a TTY): ask the user to run it in their own terminal, then continue once it's done. The same applies todatabricks auth loginwhen no browser can open (headless or sandboxed sessions). - Service principal (M2M): client id/secret via profile or env. Use for CI/automation; never a personal PAT in CI.
- In-platform (notebook/cluster):
DATABRICKS_HOST/DATABRICKS_TOKENare already injected, so no setup is needed.
- OAuth U2M (default, interactive):
- Confirm before writing any profile; auth writes to
~/.databrickscfg. - Verify:
databricks current-user me --profile <name>returns the expected user. For account-level profiles,current-user medoesn't exist; usedatabricks auth describe --profile <name>and check the resolved host and account ID.
Never echo tokens or secrets back. Never auto-select a profile. When done,
suggest /databricks:doctor for a full health check.
Sponsors
Details
- Marketplace
- databricks/databricks-agent-skills
- Source
- plugins/databricks/claude
- Category
- Development