Databricks skills for the CLI, Apps, Lakebase, Model Serving, Lakeflow Jobs, Spark Declarative Pipelines, Declarative Automation Bundles (DABs), and classic-to-serverless migration.

GitHub stars
404

Install

/plugin install databricks@databricks-agent-skills

Run it inside Claude Code after adding the marketplace with /plugin marketplace add databricks/databricks-agent-skills.

Contents

Discovered from source

Skills 38

Databricks AI Functions

Official Docs: https://docs.databricks.com/large-language-models/ai-functions Individual function reference: https://docs.databricks.com/sql/language-manual/sql-ref-functions-builtin

Overview

Databricks AI Functions are built-in SQL and PySpark functions that call Foundation Model APIs directly from your data pipelines — no model endpoint setup, no API keys, no boilerplate. They operate on table columns as naturally as UPPER() or LENGTH(), and are optimized for batch inference at scale.

Always prefer a task-specific function over ai_query. Reach for ai_query only when no task function fits (custom/external endpoints, multimodal, or JSON beyond ai_extract's limits). Every function below shares a baseline: DBR 15.1+ (notebooks) / 15.4 ML LTS (batch), not on SQL Warehouse Classic, and region must support AI Functions — the Prereqs column lists only what's additional.

Cost & speed — each call is an LLM inference (slow and billed per token). Run a function once per row and persist the result to a Delta table; never re-invoke it on every downstream query. In demos, avoid generating tables with millions of rows — sample the input when needed so the demo runs quickly. Materialize once, then query the cheap Delta output.

The Function column links to the in-repo deep reference (full options, schemas, examples); Docs links to the official page.

FunctionTaskInputOutputExtra prereqsDocs
ai_analyze_sentimentSentiment scoringcontent STRINGSTRING — positive/negative/neutral/mixed, or NULL—↗
ai_classifyFixed-label routingcontent STRING|VARIANT, labels (2–500), [options MAP]v2.1 VARIANT — {response:[{value:label}], error_message, metadata}—↗
ai_decideEvaluate questions (boolean / choice / score) over text or structured datastate STRING|VARIANT, questions STRING (JSON), [options MAP]VARIANT — {response:{answers}, metadata, error_message}Beta; admin-gated via Previews↗
ai_extractEntity / field extractioncontent STRING|VARIANT, schema STRING (JSON), [options MAP]v2.1 VARIANT — {response:{field:{value:…}}, error_message, metadata}≤256 fields, ≤12 nesting levels↗
ai_fix_grammarGrammar correctioncontent STRINGSTRING (corrected)—↗
ai_genFree-form generationprompt STRINGSTRING—↗
ai_maskPII redactioncontent STRING, labels ARRAY<STRING>STRING (entities → [MASKED])—↗
ai_similaritySemantic similarityexpr1 STRING, expr2 STRINGFLOAT (0.0–1.0)—↗
ai_summarizeSummarizationcontent STRING, [max_words INT] (0 = uncapped)STRINGPublic Preview; English-tuned↗
ai_translateTranslationcontent STRING, to_lang STRINGSTRINGLangs: en, fr, de, hi, it, pt, es, th↗
ai_parse_documentParse PDF / Office / imagescontent BINARY, [Map('version','2.0', …)]VARIANT — pages, elements, error_statusDBR 17.3+; ≤500 pages / 100 MB↗
ai_prep_searchRAG chunking from parsed docs or textparsed VARIANT|STRING, [options MAP]VARIANT — {document:{contents, pages, source_uri}, error_status}DBR 18.2+ (serverless env v3+)↗
ai_queryHosted model service or custom serving endpoint, multimodal, complex JSON (last resort)endpoint STRING, request STRING|STRUCT, [returnType], [failOnError BOOL], [modelParameters STRUCT], [responseFormat STRING], [files]Parsed response; with failOnError => false a STRUCT{response, errorMessage}Pro/Serverless warehouse; EXECUTE on model service or CAN QUERY on serving endpoint↗
ai_forecastTime series forecasting (table-valued)observed TABLE, horizon, time_col, value_col, [group_col], [prediction_interval_width], [frequency], [seed], [parameters]Rows: time/group cols + per value {v}_forecast, {v}_upper, {v}_lower (DOUBLE)Pro/Serverless warehouse; Public Preview↗

Models run under Apache 2.0 or LLAMA 3.3 Community License — you are responsible for compliance.

Patterns

Pin extraction and classification to v2.1 in new code. Every extracted scalar and returned label has a value field, even when citations, confidence scores, and rationales are disabled. Existing v2.0 queries remain supported; update downstream projections to read value when upgrading. See the task-function reference for the version-specific contracts.

Chain task functions to enrich a column in one pass. ai_classify/ai_extract return a VARIANT — read it with the colon operator (:response):

SELECT id,
  ai_analyze_sentiment(content)                                                   AS sentiment,
  ai_summarize(content, 30)                                                       AS summary,
  ai_classify(content, '["technical","billing","other"]', map('version','2.1')):response[0].value::STRING AS category,
  ai_extract(content, '["product","error_code","date"]', map('version','2.1')):response:product:value::STRING AS product,
  ai_fix_grammar(content)                                                         AS content_clean
FROM raw_feedback;

In PySpark, call any of these inside expr(...): df.withColumn("category", expr("ai_classify(content, '[\"a\",\"b\"]', map('version','2.1')):response[0].value::STRING")) — and read extracted scalar fields via selectExpr("col:response:field:value::STRING AS field").

PII redaction before storage — ai_mask(content, ARRAY(entity_types)) returns text with entities → [MASKED].

SELECT ai_mask(message, array('person','email','phone','address')) AS message_safe FROM raw_messages;

Semantic matching / dedup — ai_similarity returns 0–1; self-join and threshold:

SELECT a.id, b.id, ai_similarity(a.name, b.name) AS score
FROM companies a JOIN companies b ON a.id < b.id
WHERE ai_similarity(a.name, b.name) > 0.85;

Forecasting — table-valued; one row per future period (+ per group). Full param/group/interval forms → 3-ai-forecast.md:

SELECT * FROM ai_forecast(
    observed => TABLE(SELECT date, sales FROM daily_sales),
    horizon => '2026-12-31', time_col => 'date', value_col => 'sales');
-- Returns: date, sales_forecast, sales_upper, sales_lower

Nested field extraction — ai_extract supports arrays of objects, including invoice line items. See extraction schemas and versioned outputs.

Document parsing (ai_parse_document) and RAG chunking (ai_prep_search) get their own staged pipeline below.

Document Processing Pipeline

Chain AI Functions stage-by-stage into Delta tables for batch document processing. The example is written as a Spark Declarative Pipeline (SDP / Lakeflow / DLT) — CREATE OR REFRESH STREAMING TABLE with STREAM(...) sources. To run the same logic standalone in a notebook / SQL warehouse, swap each CREATE OR REFRESH STREAMING TABLE x AS for CREATE OR REPLACE TABLE x AS and drop the STREAM(...) wrappers. In SDP Python it's @dp.table with from pyspark import pipelines as dp.

Pass the parsed VARIANT directly into extraction and classification to preserve layout and source context. For plain-text or Markdown files, skip parsing: use read_files(..., format => 'text', wholetext => true) and pass its value column. Without wholetext => true, text input is read one line per row.

-- Stage 1 — parse supported PDF, image, or Office documents, filter parse errors
CREATE OR REFRESH STREAMING TABLE raw_parsed AS
SELECT path,
  parsed AS parsed_content,
  parsed:error_status AS parse_error
FROM (
  SELECT path, ai_parse_document(content, map('version','2.0')) AS parsed
  FROM STREAM read_files('/Volumes/my_catalog/doc_processing/landing/', format => 'binaryFile')
)
WHERE is_variant_null(parsed:error_status);

-- Stage 2 — classify document type (cheap, no endpoint selection)
CREATE OR REFRESH STREAMING TABLE classified_docs AS
SELECT *,
  ai_classify(parsed_content, '["invoice","purchase_order","receipt","contract","other"]', map('version','2.1')):response[0].value::STRING AS doc_type
FROM STREAM raw_parsed;

-- Stage 3 — extract fields; ai_extract returns a VARIANT, read fields with `:`
CREATE OR REFRESH STREAMING TABLE extracted AS
SELECT path, doc_type,
  result:response:invoice_number:value::STRING AS invoice_number,
  result:response:vendor_name:value::STRING    AS vendor_name,
  result:response:total_amount:value::DOUBLE   AS total_amount,
  result:error_message::STRING           AS extract_error
FROM (
  SELECT *, ai_extract(parsed_content,
    '{"invoice_number":{"type":"string"},"vendor_name":{"type":"string"},"total_amount":{"type":"number"}}',
    map('version','2.1')) AS result
  FROM STREAM classified_docs WHERE doc_type = 'invoice' AND parsed_content IS NOT NULL
);

In a batch job, retain per-row errors for inspection or route them to a sidecar table: check ai_extract's result:error_message with is_variant_null for JSON null on success, and for ai_query pass failOnError => false and check ai_response.errorMessage (a STRUCT field, dot-accessed). SQL IS NULL does not recognize a JSON null inside VARIANT. See 2-ai-query.md.

Custom RAG Pipeline — Parse → Prep → Index

For retrieval rather than field extraction: ai_parse_document → ai_prep_search (semantic chunking + context enrichment, DBR 18.2+) → Vector Search Delta Sync index. ai_prep_search returns chunk_id, chunk_to_retrieve, and chunk_to_embed (enriched with title/headers/page) — embed chunk_to_embed, return chunk_to_retrieve to the LLM. Shown standalone; in an SDP swap CREATE OR REPLACE TABLE for CREATE OR REFRESH STREAMING TABLE + STREAM read_files(...).

CREATE OR REPLACE TABLE parsed_chunks AS
WITH prepped AS (
  SELECT path AS source_path, ai_prep_search(ai_parse_document(content, map('version','2.0'))) AS prep
  FROM read_files('/Volumes/my_catalog/doc_processing/docs/', format => 'binaryFile')
)
SELECT
  variant_get(chunk, '$.chunk_id',          'STRING') AS chunk_id,
  variant_get(chunk, '$.chunk_to_retrieve', 'STRING') AS chunk_to_retrieve,
  variant_get(chunk, '$.chunk_to_embed',    'STRING') AS chunk_to_embed,
  source_path
FROM prepped LATERAL VIEW explode(variant_get(prep, '$.document.contents', 'ARRAY<VARIANT>')) c AS chunk;

Then enable CDF (ALTER TABLE parsed_chunks SET TBLPROPERTIES (delta.enableChangeDataFeed = true)) and use the databricks-vector-search skill to build a Delta Sync index: PK chunk_id, embedding source chunk_to_embed, return chunk_to_retrieve.

Beyond batch:

  • Ask questions over the output — point a Genie Agent at the resulting Delta table for natural-language querying instead of hand-written SQL; see the databricks-genie-agents skill.
  • Low-latency / serving — to expose this as a real-time, governed endpoint (e.g. register a model to Unity Catalog and serve it), use the databricks-model-serving skill.
  • Production incremental ingestion — for a runnable end-to-end streaming ai_parse_document job (checkpoints, trigger(availableNow=True)), see databricks/bundle-examples · job_with_ai_parse_document.

Reference Files

  • 1-task-functions.md — Deep reference for every task-specific function: full options/schemas (e.g. ai_extract v2.1 citations + confidence scores, ai_classify multilabel, ai_parse_document options + output schema, ai_prep_search chunk schema) and non-trivial examples. The Overview table above links to each function's section directly.
  • 2-ai-query.md — ai_query complete reference: all parameters, structured output with responseFormat, multimodal files =>, UDF patterns, and error handling
  • 3-ai-forecast.md — ai_forecast parameters, single-metric, multi-group, multi-metric, and confidence interval patterns

Common Issues

IssueSolution
ai_parse_document not foundRequires DBR 17.3+. Check cluster runtime.
ai_prep_search not foundRequires DBR 18.2+ (serverless env v3+).
ai_decide not found / unavailableBeta — a workspace admin must enable it on the Previews page. Requires Databricks SQL (not SQL Classic) or DBR 15.4 LTS+.
explode() fails on a VARIANTexplode needs ARRAY — cast first: explode(variant_get(prep, '$.document.contents', 'ARRAY<VARIANT>')).
Embedding the wrong RAG columnEmbed chunk_to_embed (context-enriched); return chunk_to_retrieve to the LLM.
ai_forecast failsRequires Pro or Serverless SQL warehouse — not available on Classic or Starter.
All functions return NULLInput column is NULL. Filter with WHERE col IS NOT NULL before calling.
ai_translate fails for a languageSupported (8): English (en), French (fr), German (de), Hindi (hi), Italian (it), Portuguese (pt), Spanish (es), Thai (th) — to_lang takes the code or the full name (see references/1-task-functions.md). Use ai_query with a multilingual model for others.
ai_classify returns unexpected labelsUse clear, mutually exclusive label names. Fewer labels (2–5) produces more reliable results.
ai_query raises on some rows in a batch jobAdd failOnError => false — returns a STRUCT with .response and .errorMessage (dot-accessed) instead of raising.
ai_query fails with 403 "no longer available"Discover the exact replacement system.ai model service using the AI Gateway listing; legacy names do not always have a matching model service. Custom serving endpoints keep their existing names. See 2-ai-query.md.
Batch job runs slowlyUse DBR 15.4 ML LTS cluster (not serverless or interactive) for optimized batch inference throughput.

Databricks AI Runtime (databricks air)

Databricks AI Runtime is a set of databricks air CLI commands for submitting GPU training workloads to Databricks serverless compute. It manages environment setup, distributed training configuration, and workload lifecycle, without requiring you to manage clusters or infrastructure.

This skill covers the three everyday flows: starting a run, checking on a run, and monitoring a run as it trains.

Reference discipline

The CLI's built-in help is the source of truth for field names, accelerator types, defaults, and constraints. Prefer it at runtime over anything memorized. Look up a config field by passing its path to -h on the run command (the path is a separate argument):

databricks air run -h config                              # list the YAML fields
databricks air run -h config.compute                      # one field
databricks air run -h config.compute.accelerator_type     # a nested field

databricks air <subcommand> --help documents each command's flags. This skill covers workflow and the few things the help does not spell out.

Session setup

Check the CLI version, confirm the command is available, and list your Databricks config profiles:

databricks version
databricks air --help
databricks auth profiles --skip-validate    # available Databricks config profiles

The examples use flattened code_source fields, available in Databricks CLI 1.20.0 and newer. On an older CLI, use the equivalent nested snapshot format shown under Workload YAML. CLI 1.20.0 and newer accept both formats. Do not mix nested and flattened fields in the same configuration.

Pass -p <profile> on every command. Add -o json for scripted/programmatic calls so you get a structured envelope instead of human text:

  • Success: {"v": 1, "ts": "...", "data": {...}}
  • Error: {"v": 1, "ts": "...", "error": {"code": "...", "kind": "...", "message": "...", "retryable": bool}}

Config validation (input) errors are the exception: they print as plain text to stderr with a non-zero exit even under -o json, so check the exit code and stderr too, not just the parsed JSON. When you do get a structured error, its kind tells you what to do:

kindMeaningAction
PERMANENTNot authenticated / permission denied / feature disabled / non-retryableIf auth: databricks auth login --host <workspace-url>
NOT_FOUNDBad run ID / resourceCheck the input
CONFLICTIdempotency key collisionUse a new --idempotency-key
TRANSIENTRetryableRetry

Subcommands

CommandWhat it does
databricks air runSubmit a workload from a YAML file. Add --watch to stream logs until it finishes, --dry-run to validate without submitting, and --override <field>=<value> (repeatable) to change a field without editing the YAML.
databricks air get <JOB_RUN_ID>Show status, config, timing, and the failure reason for one run.
databricks air listList your active runs. A run that just finished drops out of the default view; add --all-status to include finished runs. Also --all-users, --filter KEY=VALUE, --limit.
databricks air cancel <JOB_RUN_ID> [<JOB_RUN_ID>...] / ... --allCancel one or more runs, or all of your active runs (--all prompts; add --yes in scripts).
databricks air logs <JOB_RUN_ID>Print a run's logs. Newer CLIs print the logs available now and exit, and stream with --follow; older CLIs (no --follow in --help) stream an active run by default.
databricks air pools list / pools get <POOL_ID>View the provisioned GPU pools available to the workspace.
databricks air images pushPush a local or remote container image to Databricks Artifact Registry. See docker-images.md.
databricks air convert-to-dabs <yaml_path>Convert a workload YAML into a Databricks Asset Bundle (local only; deploy with databricks bundle deploy).

For flags and defaults, run databricks air <subcommand> --help.

Workload YAML

experiment_name: my-training-job     # alphanumeric, - and _ only
compute:
  num_accelerators: 1
  accelerator_type: GPU_1xA10        # run `databricks air run -h config.compute` for valid values
environment:
  version: "6"                       # Standard environment; use databricks_ai_v6 for PyTorch preinstalled
  dependencies:
    - torch
    - mlflow
code_source:
  root_path: "."                     # relative to workload.yaml; extracted to $CODE_SOURCE_PATH on each node
max_retries: 0                       # while iterating; see Retries below
command: |-
  cd "$CODE_SOURCE_PATH"
  python train.py

Submit with databricks air run --file workload.yaml -p <profile>. Field details live in databricks air run -h config.<field> (the authoritative source).

For an older CLI, replace only the code_source block above with:

code_source:
  type: snapshot
  snapshot:
    root_path: "."

The values and behavior are the same: add type: snapshot under code_source and put all snapshot fields (root_path, and any remote_volume, git, or include_paths) inside code_source.snapshot. Keep the rest of the workload unchanged. Check databricks air run -h config.code_source for the fields supported by the installed CLI.

Accelerator types

Run databricks air run -h config.compute to see the accelerator types. On CLI 1.23.0 and newer it lists the types available in the current workspace and how many GPUs each type has per node. If it cannot fetch them, it prints a warning. Some older CLI versions also show a built-in fallback that may include types the workspace can't use. Do not guess a type from memory: availability depends on the workspace, cloud, and region, and matching is case-sensitive.

Type names follow GPU_<GPUs per node>x<GPU>, for example GPU_1xA10, GPU_1xH100, GPU_8xH100, and GPU_8xB300. num_accelerators is the total GPU count across all nodes, so it must be a multiple of the per-node count: 8 on an 8x type is one node, 16 is two nodes. GPU_1xH100 does not support multi-node configurations, so set num_accelerators: 1.

Python environment

Pick the environment from what the training code imports, before you submit, so a missing package does not surface only after a GPU has been provisioned:

environmentWhat it providesUse when
version: "6" (the default)Standard environment: Python and CUDA only. Since version 5, it does not include torch or torchvision.You control the dependency stack. List every import, including torch, under dependencies.
version: databricks_ai_v6Databricks AI environment: PyTorch, Transformers, and other ML frameworks preinstalled.Typical PyTorch or Hugging Face training. Add extras under dependencies.
unity_catalog_image: <catalog>.<schema>.<image>:<tag>Your own Docker image.System libraries or CUDA extensions; see Custom Docker images.

Before submitting, check the script's imports against the chosen environment. For the exact packages in each environment version, see the environment version release notes. Do not combine unity_catalog_image with version or dependencies.

Configuring the workload

These optional top-level fields cover the common needs. Run databricks air run -h config.<field> for the full reference on any of them.

Hugging Face gated models

When a model needs a Hugging Face (HF) token, lead with this checklist before asking for other training details or running secrets commands:

  1. Accept the model license on HF and, if required, wait for access approval. Create a fine-grained read-only token with Read access to contents of all public gated repos you can access. For a private repo, also grant read access to that specific repo.
  2. Choose an existing scope the user owns and can write. If they need to find one, have them filter list-scopes to their own username or scopes they created, then confirm ownership and access; a name match alone is not proof. For example, the user can run the following with their own username:
    databricks secrets list-scopes -p <profile> -o json | jq -r --arg user "<your-username>" '.[] | select(.name | contains($user)) | .name'
    
    Never browse or inspect other users' scopes or keys. Create a scope only if needed; if create-scope reports Max limit of 2000 scopes has been reached, reuse an existing user-owned scope instead. The user may need to run list-acls on their own scope to check access. Before using <scope> in any databricks secrets ... command, validate it: it must be an existing scope the user owns, from their own scopes, and match the scope-name charset ^[A-Za-z0-9_.@-]+$. If it doesn't, have the user run the commands themselves. Always double-quote it. If the scope is Azure Key Vault-backed (backend_type is AZURE_KEYVAULT), put-secret can't write to it: have the user add the token to that vault with their own tooling and use that secret's name as the key below.
  3. First check whether hf-token already exists in the scope, using the key-name check in step 4. If it does, reuse it only if it is the right token; otherwise pick a new key such as hf-token-gemma, or get the user's explicit OK before overwriting it, since other workloads may read it. Use the chosen key in steps 4 and 5. Have the user run this in their own terminal and enter the token at the hidden prompt. Never paste the token into chat or a command line.
    databricks secrets put-secret "<scope>" hf-token -p <profile>
    
  4. Once they have stored it and named their scope, verify only the key name in that one scope, filtering the output to hf-token:
    databricks secrets list-secrets "<scope>" -p <profile> -o json | jq -r '.[] | select(.key == "hf-token") | .key'
    
  5. Reference secrets: {HF_TOKEN: "<scope>/hf-token"} in the workload. Quote the reference, particularly when the scope contains @ or .. Transformers reads HF_TOKEN automatically.

Gemma 3 is gated; Mistral Small 3.2, gpt-oss-20b, and Qwen3 are ungated and do not need a token just to download their public weights. Check the specific model's HF page for current access requirements.

Training data in Unity Catalog

When the training script reads rows from a Unity Catalog (UC) table, use UCDataset (Beta), a PyTorch dataset that reads the table's files directly. An AIR workload has no spark global, so code that calls spark.table() fails. Lead with this checklist before writing the script:

  1. Use environment.version: databricks_ai_v6 and add databricks-uc-dataset==0.0.1b1 to environment.dependencies. If the script also imports databricks.air.data, for example its DataLoader, add "databricks-sdk-air[data]" as well, as the AIR docs ask of CLI workloads; some databricks_ai_v6 builds don't include it. With unity_catalog_image, which can't be combined with version or dependencies, install these packages in the image instead.
  2. Launch with torchrun, even on one GPU, for example torchrun --nproc_per_node=$LOCAL_WORLD_SIZE train.py. AIR sets WORLD_SIZE but not RANK, and UCDataset raises when only one of them is set. With DDP, call torch.distributed.init_process_group() before you create the dataset, and create it on every rank.
  3. The workload reads the table as the identity that submits it, which needs USE CATALOG, USE SCHEMA, and SELECT. UCDataset reads only Delta tables in UC. It can't read views, materialized views, streaming tables, shallow clones, tables with a VARIANT column, or tables with row filters or column masks; see the limitations.
  4. Before submitting, check each table, so a wrong name, a missing privilege, or an unsupported table fails in seconds instead of after the GPUs start. Keep the arguments in single quotes so the shell doesn't expand them:
    databricks tables get '<catalog>.<schema>.<table>' -o json -p <profile>
    databricks experimental aitools tools query 'SELECT COUNT(*) FROM IDENTIFIER(:t)' --param 't=<catalog>.<schema>.<table>' -p <profile>
    
    Continue only if table_type is MANAGED or EXTERNAL, data_source_format is DELTA, there's no row_filter, no column has a mask or a VARIANT type, and the count returns. Backtick-quote name parts that contain characters other than letters, digits, and underscores; if a name contains ', have the user run the checks. The query needs a SQL warehouse (add --warehouse <id> to choose one), and experimental commands can change between CLI versions. If the query can't run for a reason unrelated to the table, report the check as inconclusive, not as a missing privilege.

If creating the dataset fails with PERMISSION_DENIED: ... prohibited from retrieving temporary storage credentials from Unity Catalog in network restricted workspaces, the workspace's serverless network policy blocks UCDataset. The checks in step 4 still pass in that case, and tables in the samples catalog may still load. Retrying, or changing how the script calls UCDataset, won't help: tell the user, since an account admin controls that policy. If the user says the policy changed and the same error comes back, resubmit at most once with max_retries: 0, then stop.

For files in a UC volume, such as images or audio, use databricks.air.data.UCVolumeDataset instead. For a complete workload, see Load UC tables into PyTorch with UCDataset and the cookbook's click-through-rate recipe.

Environment variables and secrets

Set plain values under env_variables; pull sensitive values from Databricks Secrets under secrets (each entry maps an env var name to a scope/key, and the value is materialized on the worker, never logged):

env_variables:
  BATCH_SIZE: "32"
secrets:
  HF_TOKEN: "<scope>/hf-token"          # -> $HF_TOKEN on every node
  WANDB_API_KEY: "<scope>/wandb_key"

The scope and keys must already exist. Prefer a user-owned scope; create one only if needed, and have the user store secret values in their own terminal. Secret env var names must not collide with env_variables names.

Parameters (hyperparameters)

parameters is the clean way to pass hyperparameters into your script and change them between runs without editing training code. The CLI writes them to a YAML file on the cluster and exposes its path as $HYPERPARAMETERS_PATH:

parameters:
  learning_rate: 2e-5
  batch_size: 32
  warmup_steps: 1000
  use_amp: true

Read them in the script, converting every number to the type your code expects:

import os, yaml
with open(os.environ["HYPERPARAMETERS_PATH"]) as f:
    hp = yaml.safe_load(f)
lr = float(hp["learning_rate"])
batch_size = int(hp["batch_size"])
warmup_steps = int(hp["warmup_steps"])
use_amp = hp["use_amp"]

Always convert numeric values with float() or int(), including values in nested dicts and lists: yaml.safe_load can return floats such as 2e-5, 0.00002, or 1e6 as strings. Write integers as plain digits (1000000, not 1e6), because int("1e+06") fails.

Strings, booleans, lists, and nested dicts arrive with their types. Quote a value that must stay a string but looks like a number (run_tag: "1e5", version: "1.10"); unquoted, it is read as a number. To sweep a hyperparameter without editing the YAML, resubmit with --override parameters.learning_rate=0.0005 (and a fresh idempotency key).

Retries

Failed attempts are retried automatically: max_retries defaults to 3 when unset (confirm with databricks air run -h config.max_retries). Each retry provisions GPUs and reruns the whole command, so a deterministic failure (an import error, a typo, a bad path) fails 4 times before the run gives up.

Set max_retries: 0 while writing, debugging, or iterating on a workload, either in the YAML or with --override max_retries=0. Raise it only for a long, stable training run where retrying can recover from transient infrastructure failures, and make the training code resume from a checkpoint so a retry does not start over. When timeout_minutes is set, it applies per attempt.

Permissions

By default only you can see the run. Grant others access to the submitted job with permissions (DABs-compatible format); levels are CAN_VIEW, CAN_MANAGE_RUN, CAN_MANAGE:

permissions:
  - group_name: data-engineering
    level: CAN_VIEW
  - user_name: alice@example.com
    level: CAN_MANAGE

Other fields

FieldUse
timeout_minutesWall-clock limit per attempt, in minutes.
idempotency_tokenYAML equivalent of --idempotency-key: resubmitting with the same token returns the existing run.
mlflow_run_name, mlflow_experiment_directoryName the MLflow run, or choose the workspace directory that holds the experiment.
compute.pool_id, compute.priority_classRun on a pre-provisioned GPU pool (find IDs with databricks air pools list). priority_class is NORMAL or BEST_EFFORT (preemptable) and requires pool_id.
containersRun different images and commands on different node ranks, for example a trainer and an inference server. Replaces command; each entry needs name, command, ranks, and unity_catalog_image. See databricks air run -h config.containers.

Environment variables provided on each node

databricks air sets these automatically on every node; use them in command: and in your training code.

VariableMeaning
CODE_SOURCE_PATHAbsolute path to your extracted code_source on the node (cd $CODE_SOURCE_PATH).
MASTER_ADDR / MASTER_PORTAddress/port of the rank-0 node for distributed coordination.
NUM_NODESTotal number of nodes.
NODE_RANKRank of the current node (0 to NUM_NODES-1). Also set as POD_RANK.
WORLD_SIZETotal number of GPUs across all nodes.
LOCAL_WORLD_SIZENumber of GPUs on this node.
MLFLOW_RUN_IDMLflow run created for this attempt (log metrics/artifacts to it). mlflow.start_run() attaches to it. The Hugging Face Trainer's MLflow integration starts it automatically, so don't also call mlflow.start_run() around trainer.train(): it fails with "Run ... is already active".
HYPERPARAMETERS_PATHPath to the parameters YAML file (only set when parameters: is defined).

Your command: runs once per node, so per-process ranks (RANK, LOCAL_RANK) are not set by the platform. Your distributed launcher (torchrun, accelerate) assigns them when it spawns one process per GPU; pass it $NODE_RANK / $NUM_NODES / $MASTER_ADDR / $MASTER_PORT.

Starting a run

Start from a cookbook recipe when one fits. The AI Runtime Cookbook has runnable recipes, each a workload.yaml plus source code, grouped by domain (language-models/, computer-vision/, recommendation-and-ranking/, time-series/, multimodal/, serving/, and more; Ray recipes live in these domain folders with a ray- prefix). getting-started/air-cli/ is the minimal first run. Adapting the closest recipe is usually faster and less error-prone than writing a workload from scratch:

git clone https://github.com/databricks-solutions/ai-runtime-cookbook.git
cd ai-runtime-cookbook/<domain>/<recipe>     # e.g. getting-started/air-cli
# Review workload.yaml (and the recipe README) for values to configure, then:
databricks air run --file workload.yaml --dry-run -o json -p <profile>
databricks air run --file workload.yaml --watch \
  --idempotency-key <key> -o json -p <profile>  # key as in step 5

Apply the steps below to the recipe's YAML the same way as to a new one, including the stable --idempotency-key from step 5.

  1. Turn the user's intent into the YAML. Extract model, dataset path, GPU count and type, and the training command from the request. For gated models, lead with the token checklist first. For data in a UC table, follow the UC data checklist. Ask before synthesizing a config when a remaining critical detail is ambiguous (data path, checkpoint dir, custom image).

  2. Write the command: using the env vars above. Common launch shapes:

    Frameworkcommand: shape
    plain torch, single GPUpython train.py (with UCDataset, use torchrun; see the UC data checklist)
    torch, multi-GPU (1 node)torchrun --nproc_per_node=$LOCAL_WORLD_SIZE train.py
    torch, multi-nodetorchrun --nnodes=$NUM_NODES --nproc_per_node=$LOCAL_WORLD_SIZE --node_rank=$NODE_RANK --master_addr=$MASTER_ADDR --master_port=$MASTER_PORT train.py
    HuggingFace Accelerateaccelerate launch --config_file accelerate.yaml train.py
    DeepSpeeddeepspeed --num_gpus=$LOCAL_WORLD_SIZE train.py
    Composer / MosaicMLcomposer train.py

    Multi-line commands go under command: |-. Access uploaded code via $CODE_SOURCE_PATH. Total GPUs = nodes x accelerators-per-node; see Accelerator types.

  3. Pre-flight the referenced resources so the job does not die on startup:

    • Every package the script imports is in the chosen environment or its dependencies (the Standard environment has no torch).
    • max_retries: 0 is set while the workload is new or being debugged.
    • For each secrets reference, verify only its key name in the user's chosen scope, filtering list-secrets as in the token checklist. Never browse other users' scopes or list their keys.
    • Every UC table the script reads passes both checks in the UC data checklist. A dry run doesn't check UC tables, however the script gets their names.
    • If environment.unity_catalog_image is set, the image has been pushed to Databricks Artifact Registry in the same workspace and the submitting identity can read it (see docker-images.md).
  4. Dry-run before submitting. This validates the config without launching:

    databricks air run --file workload.yaml --dry-run -o json -p <profile>
    

    On CLI 1.20.0 and newer, the dry run also checks the config against the workspace, so it needs a working profile. A config-validation error is a hard stop: it prints as plain text on stderr with a non-zero exit, so show that message to the user and fix the YAML. Other failures are infra/auth and should bubble up.

  5. Submit. Use a stable --idempotency-key so an accidental re-run does not double-submit (compose it from <user>-<experiment>-<YYYYMMDD>-v<N> and bump vN on each resubmit; avoid dynamic subexpressions like $(uuidgen) so the command stays auto-approvable). If you want to wait for the run to finish or stream live progress, add --watch:

    databricks air run --file workload.yaml --watch \
      --idempotency-key <key> -o json -p <profile>
    

    --watch streams logs and status updates until the run finishes and exits 0 only if the run succeeds. With -o json, it emits JSONL and includes the run ID. For long runs, use the agent's background-execution support. Stopping --watch does not cancel the run, so if it exits non-zero after returning a run ID, check air get before resubmitting. If the user only wants to submit the run, omit --watch and return the run ID.

Checking on a run

databricks air get <JOB_RUN_ID> -o json -p <profile>        # status, config, timing, failure reason
databricks air list -o json -p <profile>                    # your active runs
databricks air list --all-status -o json -p <profile>       # include finished runs
databricks air logs <JOB_RUN_ID> -p <profile>               # view logs for an existing run

Read the status from .data.status in the air get -o json envelope. The terminal statuses are SUCCESS, FAILED, TIMEDOUT, CANCELED, INTERNAL_ERROR, and SKIPPED. An empty air list means you have no active runs, not that the run is gone; use --all-status or air get to see a finished run.

If no waiter is active and you need periodic status snapshots, read .data.status, sleep between calls, set a hard time limit, and stop on any terminal status or command error. Because air run --watch waits for the run to finish, you do not need to poll while it is active.

Monitoring a run

Wait and stream logs. If you want to wait for a new run and receive live updates, use --watch:

databricks air run --file workload.yaml --watch -p <profile>

For a run that is already going, check whether databricks air logs supports --follow:

databricks air logs --help 2>&1 | grep -qF -- '--follow'

If that command succeeds, air logs prints the logs available now and exits, which is the right way to check progress, and --follow streams:

databricks air logs <JOB_RUN_ID> --tail 100 -p <profile>   # print the last 100 lines and exit
databricks air logs <JOB_RUN_ID> --follow -p <profile>     # stream until the run finishes
# --node N (one node), --tail N (last N lines), --minutes N (last N minutes),
# --retry N (one retry attempt, 0-indexed), --download-to DIR (save all nodes' logs)

If the check fails, the CLI is older: air logs <JOB_RUN_ID> streams an active run until it finishes, and --follow is rejected. To check progress without blocking on an older CLI, use databricks air get <JOB_RUN_ID>.

Loss, GPU utilization, and metric history live in MLflow. The CLI streams logs and status, not training metrics. The submit/status response includes the MLflow experiment and run URL; echo it so the user can open the run in the Databricks UI for loss curves, GPU utilization, and system charts.

If a run fails

Get the failure reason, then pull the logs and surface the actual error to the user:

databricks air get <JOB_RUN_ID> -o json -p <profile>             # termination_reason
databricks air logs <JOB_RUN_ID> --tail 500 -p <profile>         # latest attempt
databricks air logs <JOB_RUN_ID> --retry 0 --tail 500 -p <profile>   # first attempt

If the run was retried, the latest attempt's logs may show a different symptom than the first failure; check --retry 0. On a multi-node run, check other nodes with --node N when node 0 looks healthy.

torchrun workloads can print hundreds of NCCL INFO lines before the user's error, so search the logs for Traceback or Error rather than reading them in order. When creating a UCDataset fails, the traceback ends with the UC error, such as TABLE_DOES_NOT_EXIST or PERMISSION_DENIED. Under DDP it's a single long rank 0 failed to build the Delta plan line. In databricks-uc-dataset 0.0.1b1, UC returned no storage location usually means the name is a view, and set both RANK and WORLD_SIZE or neither means the script wasn't launched with torchrun. See the UC data checklist. A TypeError comparing 'float' and 'str' at optimizer or scheduler setup means a hyperparameter was read without float(); see Parameters.

Show the failing traceback and let the user decide the fix. Before resubmitting, set max_retries: 0 so a deterministic error fails once.

Serving a trained model

A model trained with databricks air is served as a custom model: log it with MLflow, register it in Unity Catalog, and create a serving endpoint on CPU or GPU compute.

To serve it with the library versions you tested, without pinning torch or CUDA for the serving container, use an express deployment (Public Preview). The endpoint then runs the environment of the notebook you registered from. Express deployments require:

  • Logging and registering from an AI Runtime notebook to serve on GPU, or from a serverless notebook to serve on CPU, with mlflow>=3.12 and databricks-sdk>=0.102.0.
  • Registering with mlflow.register_model(..., env_pack="databricks_model_serving"). Do not use registered_model_name= on log_model: it does not accept env_pack and registers a standard model.
  • An environment of at most 200 GB.

For LLMs, use Custom LLM Serving, which builds on express deployments and starts an OpenAI-compatible server such as vLLM from the model's entrypoint metadata.

GPU workload_type names map to different GPUs per cloud. For example, GPU_LARGE is an L40S (48 GB) on AWS and an A100 (80 GB) on Azure. See the GPU table on the Custom LLM Serving page.

Custom Docker images

Use a custom Docker image instead of environment.version when your workload needs specific system libraries, CUDA extensions (flash-attn, apex, custom kernels), or dependencies that do not fit environment.dependencies.

Read docker-images.md for step-by-step guidance on:

  • Using Databricks-provided base images, including CUDA 13 images for B300
  • Dockerfile patterns
  • Pre-build compatibility checklist (CUDA/driver, PyTorch, NCCL, EFA/RDMA)
  • Pushing images with databricks air images push and running them with environment.unity_catalog_image
  • Troubleshooting runtime errors such as ssl.SSLError on FIPS hosts

AI/BI Dashboard Skill

Create Databricks AI/BI dashboards (formerly Lakeview dashboards). A dashboard should be showing something relevant for a human, typically some KPI on the top, and based on the story, some graph (often temporal), and we see "something happens". Follow these guidelines strictly.

Building a dashboard, in order: profile the data (Steps 1-3) → pick widgets from the Widget catalog → read 4-examples.md end to end (a complete dashboard JSON) → read the spec section of each widget you picked (catalog Spec column) and 3-filters.md for filters → build the JSON, test every query, deploy (Step 4, Step 5).

Widget catalog

Wrong version = broken widget! The #1 cause of dashboard errors: use the v below in spec.version. Spec = where the widget's exact JSON is (a section named after it): 1 1-widget-specifications.md, 2 2-advanced-widget-specifications.md, F 3-filters.md, 6 6-custom-visualizations.md. Anchored index: 1- § Widget Index.

Widget (widgetType)Use forvSpec
text (markdown)titles, section headers, notes; no spec block–1
countera KPI number, with sparkline and period-over-period delta21
tablerow-level detail, high-cardinality lists21
bar, linecomparisons across categories; trends over time (stacked/grouped, horizontal, annotations)31
pieshare of a few categories31
areatrends showing volume or stacked composition32
scatterrelationship between two measures; bubble = size by a third32
combobar + line on one chart, metrics with different scales (dual axis)12
pivotcross-tab (dimensions on rows and columns) with conditional cell styling; cohort retention32
histogramfrequency distribution of a numeric column32
heatmap"X by Y" matrix, color = aggregate32
boxdistribution (median, quartiles, outliers) across categories12
funnelstage-by-stage conversion12
waterfallpositive/negative deltas to a total (P&L bridge, MoM walk)12
sankeyflow between two or more stages12
forecast-linehistory continued by a forecast band (AI_FORECAST SQL)12
ganttschedules and events with a start and end12
symbol-maplat/lon points (bubble map)21
choropleth-mapregions (countries, states) colored by value12
path-maproutes and ordered tracks12
filter-single-select, filter-multi-select, filter-date-range-picker, filter-date-picker, filter-text-entryglobal or page-level filters2F
range-slidernumeric range filter2F
custom-vega-vizwhat standard widgets can't draw (bullet, layered, gauge, radar, sunburst)16

There is no cohort widget (use pivot with a color-scale cell style) and no Genie widget (see Linking a Genie Space).

Standard or custom? Use standard widgets by default. Use Vega-Lite when the required visualization cannot be expressed with standard widgets. A different aggregation, color palette, tooltip, reference line, or small multiple usually does not require a custom specification. Case-by-case guide: 6-custom-visualizations.md § Standard widget or custom Vega-Lite?.

For custom charts, confirm the required marks and transforms are supported by Vega-Lite. It does not support every Vega example or chart type. Keep joins and business calculations in dataset SQL; the custom specification controls how the returned data is drawn.

When a custom app fits better: dashboards are for read-only KPIs, charts, and filters over governed tables. For write-back / data entry, UI or interactions beyond the grid, embedded or auth-gated workflows, or a Genie/chat assistant as the main surface, build a Databricks App with databricks-apps (it brings in databricks-app-design). Linking an "Ask Genie" space to this dashboard stays here (see Linking a Genie Space below).

Quick Reference

TaskCommand
List warehousesdatabricks warehouses list
List tablesdatabricks experimental aitools tools query --warehouse WH "SHOW TABLES IN catalog.schema"
Get schemadatabricks experimental aitools tools discover-schema catalog.schema.table1 catalog.schema.table2
Test querydatabricks experimental aitools tools query --warehouse WH "SELECT..."
Create dashboarddatabricks lakeview create --display-name "X" --warehouse-id "WH" --dataset-catalog CATALOG --dataset-schema SCHEMA --serialized-dashboard "$(cat file.json)" --json '{"parent_path": "/Workspace/Users/<you>/path"}' (flag-only vs JSON-only fields: Step 5)
Update dashboarddatabricks lakeview update DASHBOARD_ID --dataset-catalog CATALOG --dataset-schema SCHEMA --serialized-dashboard "$(cat file.json)": re-pass both dataset flags (update replaces the dashboard; omitting them nulls the defaults and breaks every bare-table query)
Publish / unpublishdatabricks lakeview publish DASHBOARD_ID --warehouse-id WH --embed-credentials / databricks lakeview unpublish DASHBOARD_ID: --embed-credentials runs viewers' queries as the publisher so /published works without table access; resets on every publish. New dashboard: pass it unless queries must run as each viewer (RLS / individual permissions). Republish: keep the current mode (Updating).
List / get / deletedatabricks lakeview list / databricks lakeview get DASHBOARD_ID / databricks lakeview trash DASHBOARD_ID
Tag after creation (default; tracks this skill's resources)databricks workspace-entity-tag-assignments create-tag-assignment dashboards DASHBOARD_ID aidevkit_project --tag-value ai-dev-kit

--warehouse flag: if aitools tools query fails with unknown flag: --warehouse on your CLI version, drop the flag and set DATABRICKS_WAREHOUSE_ID=WH in the environment (picked up automatically).


NEW DASHBOARD CREATION WORKFLOW

You MUST test ALL SQL queries via CLI BEFORE deploying. Follow the overall logic in these steps for new dashboard - Skipping validation causes broken dashboards.

Step 1: Get Warehouse ID if not already known

# List warehouses to find one for SQL execution
databricks warehouses list

Step 2: Discover Table Schemas and existing data pattern

Know the data first: this exploration drives the Step 4 design (which widgets, filters, groupings).

Use discover-schema as the default — one call returns columns, types, sample rows, null counts, and row count. If you only know the schema, list tables first with query "SHOW TABLES IN ...".

databricks experimental aitools tools discover-schema catalog.schema.orders catalog.schema.customers

Then probe with aggregate SQL (cardinality, top values, distributions, trend grain, story confirmation) before designing; fan independent probes out in one multi-query call (--output json, check each statement's state): 7-dashboard-design.md § Profiling the data.

Dashboard queries are different: inside the dashboard JSON use bare table names (✅ FROM trips, ❌ FROM samples.nyctaxi.trips); a hardcoded prefix wins over the dataset flags and isn't portable (Dataset architecture).

Step 3: Verify Data Matches Story

Before finalizing, run the SQL Queries you intend to add in each dataset to confirm that they run properly and that the result are valid. Widgets render the query output, so the values must also make sense at a business level.

If values don't match expectations, ensure the query is correct, fix the data if you can, or adjust the story before creating the dashboard.

Step 4: Plan and build the dashboard JSON

  1. Plan the widgets: pick each one from the Widget catalog, map it to a dataset, and note which filter fields it needs (example plan).
  2. Read 4-examples.md end to end before writing any JSON. It's a complete working dashboard exercising every construct (dataset measures + MEASURE(), sparkline counters, forecast-line with annotations, pivot with conditional cells, symbol-map, histogram, range-slider filter, theme). Keep its structure; swap the tables, metrics, palette, and narrative for the user's data and story.
  3. Read the spec section of every widget you picked (catalog Spec column) for its exact JSON and version, and 3-filters.md for filters.
  4. Check the filters: every dataset you want filtered must contain the filter field (typically group by it). A pre-aggregated table without dates WON'T be date-filtered.
  5. Build the dashboard JSON as a local working file (intermediate step, not the deliverable), following the skeleton and the implementation guidelines.

Step 5: Deploy

Now deploy the JSON with databricks lakeview create (below). The task isn't complete until it succeeds and returns a dashboard ID; the JSON file is only an intermediate artifact.

Give the user the dashboard link (host: databricks auth env -o json → .env.DATABRICKS_HOST; note the v3, plain /sql/dashboards/ is the legacy path):

  • Draft: https://<host>/sql/dashboardsv3/<DASHBOARD_ID>
  • Published: append /published (only valid once you've run databricks lakeview publish DASHBOARD_ID)

After deploying, the Quick Reference lakeview commands manage the lifecycle (list, get, update, publish, unpublish, trash, tag).

# parent_path must ALREADY EXIST or create fails "Tree node ... does not exist":
databricks workspace mkdirs /Workspace/Users/me@co.com/dashboards

# Create (mix flags + --json; each field has ONE valid place):
#   --dataset-catalog / --dataset-schema: FLAG-ONLY and required. In --json the CLI warns
#     "unknown field" and drops them, so no dataset query resolves its catalog.schema.
#   parent_path: JSON-ONLY (no flag); without it the dashboard lands at /Users/<you>/<display-name>.
#   display_name / warehouse_id / serialized_dashboard: either form; flags read better.
# Capture the id with -o json and don't add 2>&1: create echoes the dashboard back and merged
# stderr breaks jq (the dashboard IS created, but you lose the id and the step looks failed).
DASHBOARD_ID=$(databricks lakeview create \
  --display-name "My Dashboard" \
  --warehouse-id "abc123def456" \
  --dataset-catalog "my_catalog" \
  --dataset-schema "my_schema" \
  --serialized-dashboard "$(cat dashboard.json)" \
  --json '{"parent_path": "/Workspace/Users/me@co.com/dashboards"}' \
  -o json | jq -r '.dashboard_id')
# Publish (--embed-credentials: see the CLI table above)
databricks lakeview publish "$DASHBOARD_ID" --warehouse-id "abc123def456" --embed-credentials

UPDATING AN EXISTING DASHBOARD

To change a dashboard that already exists, build the updated JSON with the creation workflow above, then deploy it with update + publish on the same DASHBOARD_ID — never re-run create/import. Commands: Quick Reference (update must re-pass --dataset-catalog / --dataset-schema).

  • Don't create/import to update: it mints a new id + URL and breaks shared links; create is for brand-new dashboards only.
  • update changes only the draft: /published stays on the last snapshot until you publish again, so always publish after update.
  • Keep the published credential mode: read it first (databricks lakeview get-published DASHBOARD_ID -o json | jq .embed_credentials); pass --embed-credentials only if true. Switching from individual permissions to embedded widens what viewers see: only when asked.

JSON Structure (Required Skeleton)

Every dashboard's serialized_dashboard content must follow this exact structure:

{
  "datasets": [
    {
      "name": "ds_x",
      "displayName": "Dataset X",
      "queryLines": ["SELECT col1, col2 ", "FROM my_table"]
    }
  ],
  "pages": [
    {
      "name": "main",
      "displayName": "Main",
      "pageType": "PAGE_TYPE_CANVAS",
      "layout": [
        {"widget": {/* INLINE widget definition */}, "position": {"x":0,"y":0,"width":2,"height":3}}
      ]
    }
  ]
}

Structural rules (violations cause "failed to parse serialized dashboard"):

  • queryLines: Array of strings, NOT "query": "string". Elements are joined verbatim with no separator — end each line with or \n (or strip -- comments). A line ending in -- comment with no newline swallows the next line.
  • Widgets: INLINE in layout[].widget, NOT a separate "widgets" array
  • pageType: Required on every page (PAGE_TYPE_CANVAS or PAGE_TYPE_GLOBAL_FILTERS)
  • Query binding: query.fields[].name == encodings.*.fieldName (rule). Filters bind differently: range sliders expose MIN/MAX query fields for one fieldName; parameter-bound date filters use query.parameters.

Theme & Color (always set this)

Set top-level uiSettings.theme on every dashboard (light/dark canvas, widget, font, selection colors; visualizationColors; widgetHeaderAlignment): the highest-impact polish. Two gotchas: counters take no per-widget color (value.color renders "unsupported widget definition"; the theme fontColor colors their number), and semantic colors go in color.scale.mappings as a bare hex string ({"hex": ...} and themeColorType are silently dropped on charts). Full block, theme keys, palette rules, and starter palettes: 7-dashboard-design.md § Theme & Color.

Linking a Genie Space (Optional)

For an "Ask Genie" button, add uiSettings.genieSpace (isEnabled, overrideId, enablementMode) next to theme. Genie is not a widget (no "widgetType": "assistant"). JSON: 7-dashboard-design.md § Linking a Genie Space.


Design Best Practices

Apply unless the user says otherwise: a global date filter when data has time columns, time-bounded KPIs that support period comparison (MoM, YoY), value formatting by meaning (currency, %, compact K/M/B), and chart type matched to cardinality. Details: 7-dashboard-design.md § Design best practices.

Reference Files

What are you building?Reference
Start here (read end to end before writing JSON): full working dashboard4-examples.md
Text, counter, table, bar/line, pie, symbol-map; naming, formatting, field expressions1-widget-specifications.md
Area, scatter, combo, pivot, histogram, heatmap, box, funnel, waterfall, sankey, forecast-line, gantt, choropleth / path maps2-advanced-widget-specifications.md
Dashboard with filters (global or page-level)3-filters.md
Custom Vega-Lite widgets (layered charts, bullet, gauge, radar, sunburst), standard vs custom choice6-custom-visualizations.md
Data profiling probes, theme & colors, grid layout and sizes, Genie link, data variance7-dashboard-design.md
Dataset-level measures and MEASURE()8-dataset-measures.md
Debugging a broken dashboard5-troubleshooting.md

Implementation Guidelines

1) DATASET ARCHITECTURE

  • Fewer datasets is better: one dataset backing as many widgets as possible. Clicking a chart value (a bar, a slice) filters that dataset, so every widget sharing it re-renders; narrow per-widget datasets break this cross-filtering and force explicit filter widgets. Prefer one wide dataset per domain (orders, cases, customers); split only for a different grain, pre-aggregation, or a parameter the others can't tolerate.
  • Two ways to define a dataset: SQL via queryLines (full control: WITH / JOIN / AI_FORECAST), or the UC asset shorthand {"name": "ds_x", "asset_name": "catalog.schema.table_or_view"} (tables, views, metric views; no SQL). Shapes: 8-dataset-measures.md § Defining a dataset.
  • Exactly ONE valid SQL query per dataset when using queryLines (no multiple queries separated by ;)
  • Queries must use bare table names only — no catalog, no schema prefix. Example: FROM orders, never FROM gold.orders or FROM main.gold.orders. The catalog and schema come from the --dataset-catalog and --dataset-schema flags at creation time. These flags only fill in missing parts — they do NOT override any catalog/schema written in the query.
  • SELECT must include all dimensions needed by widgets and all derived columns via AS aliases
  • Put ALL business logic (CASE/WHEN, COALESCE, ratios) into the dataset SELECT with explicit aliases
  • Add ORDER BY when visualization depends on data order:
    • Time series: ORDER BY date for chronological display
    • Rankings/Top-N: ORDER BY metric DESC LIMIT 10 for "Top 10" charts
    • Categorical charts: ORDER BY metric DESC to show largest values first
Dataset-level measures + MEASURE()

Declare reusable measures in a dataset's columns (or rely on a metric view's YAML measures, without redeclaring them) and reference them from any widget as MEASURE(`Name`) (counter, table, bar, line, pie, pivot; mixes with inline aggregations). Shapes and examples: 8-dataset-measures.md.

2) WIDGET FIELD EXPRESSIONS

CRITICAL: Field Name Matching Rule For chart, table, counter, and custom-viz widgets, each selected name in query.fields MUST exactly match its fieldName in encodings. If they don't match, the widget shows "no selected fields to visualize" error!

For custom-vega-viz the binding has extra layers (spec.encodings.fields, spec.data.queryName, Vega-Lite data.name: "databricks_query", spec as a JSON string in spec.jsonSpec.spec): read 6-custom-visualizations.md § Widget Contract before generating one.

Correct vs mismatched example: 5-troubleshooting.md § no selected fields.

Allowed expressions in widget queries (you CANNOT use CAST or other SQL in expressions):

{"name": "(sum|avg|count|countdistinct|min|max)(col)", "expression": "(SUM|AVG|COUNT|COUNT(DISTINCT)|MIN|MAX)(`col`)"}
{"name": "(daily|weekly|monthly)(date)", "expression": "DATE_TRUNC(\"(DAY|WEEK|MONTH)\", `date`)"}
{"name": "field", "expression": "`field`"}

If you need conditional logic or multi-field formulas, compute a derived column in the dataset SQL first.

Some widgets support additional expressions: Gantt can use FIRST(...) for status fields, and geometry map bindings require ST_ASGEOJSON(...). See the corresponding widget reference for these shapes.

3) SPARK SQL PATTERNS

  • Date math: date_sub(current_date(), N) for days, add_months(current_date(), -N) for months
  • Date truncation: DATE_TRUNC('DAY'|'WEEK'|'MONTH'|'QUARTER'|'YEAR', column)
  • AVOID INTERVAL syntax - use functions instead

4) LAYOUT (12-Column Grid, NO GAPS)

Every page must include "layoutVersion": "GRID_V1" alongside pageType. Positions are {"x", "y", "width", "height"}; by default each row fills width 12 exactly (no gaps). Counters height 3-4 (never 2), charts 5-6, text headers 12x1 with title and subtitle as separate widgets. Column subdivision by audience, the size table, a standard page structure, and staggered layouts: 7-dashboard-design.md § Layout.

5) CARDINALITY & READABILITY (CRITICAL)

Dashboard readability depends on limiting distinct values:

Dimension TypeMax ValuesExamples
Chart color/groups3-84 regions, 5 product lines, 3 tiers
Filters4-158 countries, 5 channels
High cardinalityTable onlycustomer_id, order_id, SKU

Before creating any chart with color/grouping:

  1. Check column cardinality via discover-schema or a COUNT DISTINCT query
  2. If >10 distinct values, aggregate to higher level OR use TOP-N + "Other" bucket
  3. For high-cardinality dimensions, use a table widget instead of a chart

Before a trend chart, check variance: a small MAX - MIN range (e.g. 83-89% on 0-100) looks flat; prefer a KPI with delta or a table (details).

6) QUALITY CHECKLIST

Before deploying, verify:

  1. All widget names use only alphanumeric + hyphens + underscores
  2. Every page has "layoutVersion": "GRID_V1"
  3. All rows sum to width=12 with no gaps
  4. KPIs use height 3-4, charts use height 5-6
  5. Chart dimensions have reasonable cardinality (≤8 for colors/groups)
  6. All widget query expressions reference existing dataset columns or aliases
  7. Visualization field bindings match exactly (charts, tables, counters, custom viz: query.fields[].name == encoding fieldName, e.g. both "sum(spend)"); validate filters against their own contracts.
  8. Counter datasets: use disaggregated: true for 1-row datasets, disaggregated: false with aggregation for multi-row
  9. Percent values must be 0-1 for number-percent format (0.865 displays as "86.5%", don't forget to set the format). If data is 0-100, either divide by 100 in SQL or use number format instead.
  10. SQL uses Spark syntax (date_sub, not INTERVAL)
  11. All SQL queries tested via CLI and return expected data
  12. Every dataset you want filtered MUST contain the filter field — filters only affect datasets with that column in their query
  13. abbreviation: "compact" needs decimalPlaces — without it the value isn't rounded (renders $9.756278496M, not $9.76M)

Related Skills

Data App Design

Make Databricks data + AI apps that communicate clearly and compile to real AppKit code. This skill merges two bodies of knowledge and binds them to implementation:

  • Composition — what to show, how much to abstract, how to lay it out → references/dashboard-patterns.md
  • Notation — make comparable things look comparable; honest scales; scenario marks → references/ibcs-notation.md
  • Implementation — the exact AppKit components, hooks, and tokens to use → references/appkit-cheatsheet.md

For package scripts and project-local CLI tools, use the parent skill's package-manager selection and command placeholders.

Design advice that doesn't name a real component is incomplete. Always end at a component plan.

When to use / when NOT

  • USE for: the data screens of a custom-code Databricks App (AppKit/React) — overview/KPI pages, reports, metric/ontology pages, variance analysis, charts, tables, and Genie/NL data surfaces — design or critique.
  • Do NOT use for: authoring managed AI/BI (Lakeview) dashboards (→ databricks-aibi-dashboards), generic frontend (forms, auth, settings, marketing), or scaffolding/build/deploy (→ databricks-apps). A plain "create a dashboard" / "build a dashboard" request (no app / AppKit / React / custom-code signal) means a managed AI/BI (Lakeview) dashboard → use databricks-aibi-dashboards, not this skill. If a request is "add a form", "deploy this", or "build a Lakeview / AI-BI dashboard", this skill should not fire.
  • Relationship: databricks-apps builds/runs the app; this skill decides what the data screens should look like and which primitives realize them.

Workflow

  1. Frame — audience, the decision/question, refresh cadence, device, primary task. One sentence.
  2. Genre — pick the closest from references/dashboard-patterns.md (static / analytic / magazine / infographic / repository / embedded mini). State it.
  3. Compose — choose content + composition patterns (data abstraction, meta-info, layout, interaction, color). Make the tradeoff explicit: what's summarized, hidden, paginated, or made interactive — and why.
  4. Apply notation — run the relevant references/ibcs-notation.md rules: message-in-title, scenario marks (actual/PY/plan/forecast), honest scales, semantic color. On any chart-vocabulary conflict, IBCS wins (see the conflict note in that file).
  5. Bind to components — map every element to a primitive that's actually exported from @databricks/appkit / @databricks/appkit-ui (see references/appkit-cheatsheet.md); never cite a component AppKit doesn't ship. There's no prebuilt KPI/trend/distribution card — compose those from primitives, following the notation rules. Use colorPalette + semantic tokens, never hardcoded hex. Bind data with useAnalyticsQuery/queryKey + sql.* params.
  6. Cover the states — every data view must handle loading / empty / error / partial (see checklist).
  7. Review — run the checklists in both reference files; lead critiques with the highest-impact comprehension or integrity issue, citing the affected component/file.

Required states & data realism (non-negotiable for data apps)

  • Loading → Skeleton; Empty → Empty with a useful next action; Error → inline message, never a blank panel; Partial/stale → show what you have + a freshness note.
  • Every KPI shows unit + period + comparison + freshness/source (mirror the metric definition; don't show a number with no provenance).
  • Large tables → server-side pagination/sort/filter, not client-side over a huge result set.
  • Long-running queries → optimistic loading + timeout/error UX.

AI / Genie surfaces (the "AI" half)

Gate: this section applies only if the app has a Genie / chat / natural-language / "ask your data" surface. For a pure dashboard / KPI / report app with no conversational input, skip this section and references/genie-ai-trust.md entirely. When it does apply, implement ALL five (code in references/genie-ai-trust.md): A Genie/chat/NL answer is only trustworthy if the user can see how it was produced and who it ran as. "Use GenieChat + a spinner" is NOT enough — for ANY Genie/chat surface, ship all five (copy the exact snippets from the reference):

  1. Identity — a /api/whoami route (real x-forwarded-email/x-forwarded-user headers) + the signed-in user in a Badge. Claim OBO only if user_api_scopes: [dashboards.genie] is wired; otherwise disclose the query runs as the app's service principal.
  2. Generated SQL — render attachments[].query in an inspectable "Generated SQL" Card; never hide how the answer was computed.
  3. Streaming/status — reflect useGenieChat().status (streaming/error), never a frozen spinner.
  4. Disclaimer — a persistent "AI-generated — verify" note per answer.
  5. Governance + states — genie() space config + a truthful execution-identity note (OBO when user-scoped, else service principal) + empty/error/ambiguous handling (Empty, Alert).

Output formats

Design proposal:

## Direction
[Genre, audience, primary task, design intent.]
## Pattern & notation choices
- Composition: [data info, meta info, layout, interaction, color]
- Notation: [message, scenario marks, scales, semantic color]
## Component plan        ← the part that makes it buildable
- [element] → [AppKit component] (queryKey/props), [token/palette], states handled
## Tradeoffs & risks
[What's summarized/hidden/paginated/interactive; overload, scale, a11y, maintenance risks.]

Critique: lead with the top comprehension/integrity issue, cite the component/file, then list findings by impact, each with the concrete fix (which component/token/state to change).

Anti-patterns

  • Producing a design memo with no component plan.
  • "Use semantic color" without naming the token/palette.
  • Naming a component AppKit doesn't export (e.g. a prebuilt KpiCard) — compose composites from published primitives instead.
  • Adding interaction, pages, or density the task doesn't need (over-engineering a mock-first app).
  • Forgetting loading/empty/error states, or KPIs with no freshness/source.

Databricks Applications — Python backends

First, confirm this skill is the right one. The default for new Databricks Apps is databricks-apps (AppKit — Node.js + TypeScript + React SDK). Load that skill first unless the user explicitly asks for a Python backend, is extending an existing Python app, or the team is Python-only. Everything below is the Python-backend alternative.

Critical Rules for Python apps (always follow)

  • MUST confirm framework choice or use Python Framework Selection below
  • MUST use SDK Config() for authentication (never hardcode tokens)
  • MUST use app.yaml valueFrom for resources (never hardcode resource IDs)
  • MUST use dash-bootstrap-components for Dash app layout and styling
  • MUST use @st.cache_resource for Streamlit database connections
  • MUST deploy Flask with Gunicorn, FastAPI with uvicorn (not dev servers)

Required Steps for Python apps

Copy this checklist and verify each item:

- [ ] Framework selected
- [ ] Auth strategy decided: app auth, user auth, or both
- [ ] App resources identified (SQL warehouse, Lakebase, serving endpoint, etc.)
- [ ] Backend data strategy decided (SQL warehouse, Lakebase, or SDK)
- [ ] Deployment method: CLI or DABs

Python Framework Selection

FrameworkBest Forapp.yaml Command
FastAPI (default)Any Python backend by default — async APIs, auto-generated OpenAPI docs, JSON-serving apps["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]
FlaskCustom REST APIs, lightweight apps, webhooks["gunicorn", "app:app", "-w", "4", "-b", "0.0.0.0:8000"]
DashProduction dashboards, BI tools, complex interactivity["python", "app.py"]
StreamlitRapid prototyping, data science apps, internal tools where the UI is a series of Python widgets["streamlit", "run", "app.py"]
GradioML demos, model interfaces, chat UIs["python", "app.py"]
ReflexFull-stack Python apps without JavaScript["reflex", "run", "--env", "prod"]

Default: FastAPI. Reach for FastAPI unless the user explicitly asks for Streamlit-style widget prototyping (Streamlit), a heavy dashboard grid (Dash), or a Gradio-style ML demo. FastAPI pairs naturally with a JS/HTML frontend or a JSON-consuming caller — the same posture databricks-apps uses on the Node side.


Quick Reference

ConceptDetails
RuntimePython 3.11, Ubuntu 22.04, 2 vCPU, 6 GB RAM
Pre-installedDash 2.18.1, Streamlit 1.38.0, Gradio 4.44.0, Flask 3.0.3, FastAPI 0.115.0
Auth (app)Service principal via Config() — auto-injected DATABRICKS_CLIENT_ID/DATABRICKS_CLIENT_SECRET
Auth (user)x-forwarded-access-token header — see references/1-authorization.md
ResourcesvalueFrom in app.yaml — see references/2-app-resources.md
SDK / Foundation Models / Model ServingUse the databricks-python-sdk skill — same WorkspaceClient and OpenAI-compatible foundation-model patterns work inside a Databricks App. For UC model services (system.ai.*), see the model-service note in references/2-app-resources.md and the databricks-unity-gateway skill
AI Search (Vector Search)Use the databricks-vector-search skill: AISearchClient(workspace_url=..., service_principal_client_id=..., service_principal_client_secret=...) from DATABRICKS_HOST (prefix https:// if missing), DATABRICKS_CLIENT_ID, and DATABRICKS_CLIENT_SECRET. Create the client and call get_index() once at startup, then reuse the index object across requests. Add databricks-ai-search>=0.83 to the app's dependencies (e.g. requirements.txt). WorkspaceClient queries are capped at a few tens of QPS
Docshttps://docs.databricks.com/dev-tools/databricks-apps/

Detailed Guides

Authorization: Use references/1-authorization.md when configuring app or user authorization — covers service principal auth, on-behalf-of user tokens, OAuth scopes, and per-framework code examples. (Keywords: OAuth, service principal, user auth, on-behalf-of, access token, scopes)

App resources: Use references/2-app-resources.md when connecting your app to Databricks resources — covers SQL warehouses, Lakebase, model serving, secrets, volumes, and the valueFrom pattern. (Keywords: resources, valueFrom, SQL warehouse, model serving, secrets, volumes, connections)

Frameworks: See references/3-frameworks.md for Databricks-specific patterns per framework — FastAPI (default), Flask, Dash, Streamlit, Gradio, Reflex — with auth integration and deployment commands. (Keywords: FastAPI, Flask, Dash, Streamlit, Gradio, Reflex, framework selection)

Deployment: Use references/4-deployment.md when deploying your app — covers Databricks CLI, Declarative Automation Bundles (DABs), app.yaml configuration, and post-deployment verification. (Keywords: deploy, CLI, DABs, asset bundles, app.yaml, logs)

Lakebase: Use references/5-lakebase.md when using Lakebase (PostgreSQL) as your app's data layer — covers auto-injected env vars, psycopg2/asyncpg patterns, and when to choose Lakebase vs SQL warehouse. (Keywords: Lakebase, PostgreSQL, psycopg2, asyncpg, transactional, PGHOST)

CLI commands: Use references/6-cli-approach.md for managing app lifecycle via CLI — covers creating, deploying, monitoring, and deleting apps. (Keywords: CLI, create app, deploy app, app logs)

Foundation Models / SDK / Model Serving: Use the databricks-python-sdk skill for the OpenAI-compatible foundation-model client, WorkspaceClient calls, and model-serving invocation — the same patterns apply inside a Databricks App. For AI Search (Vector Search) queries, use the databricks-vector-search skill instead (AISearchClient with the app's service principal; see Production and High QPS). The examples in this skill's examples/ folder (fm-minimal-chat.py, fm-parallel-calls.py, fm-structured-outputs.py, llm_config.py) show the App-side wiring only. For UC model services (system.ai.*, called on /ai-gateway/mlflow/v1), see the model-service note in references/2-app-resources.md and the databricks-unity-gateway skill.


Workflow

  1. Determine the task type:

    New app from scratch? → Load databricks-apps first (AppKit / Node). Only stay in this skill if the user explicitly asks for a Python backend. Python-backend confirmed? → Python Framework Selection — default to FastAPI. Setting up authorization? → Read references/1-authorization.md Connecting to data/resources? → Read references/2-app-resources.md Using Lakebase (PostgreSQL)? → Read references/5-lakebase.md Deploying to Databricks? → Read references/4-deployment.md Using CLI for app lifecycle? → Read references/6-cli-approach.md Calling foundation model / LLM APIs or model-serving endpoints? → Load the databricks-python-sdk skill. This skill's examples/ folder shows only the App-side wiring on top of those SDK patterns. Querying AI Search (Vector Search)? → Load the databricks-vector-search skill.

  2. Follow the instructions in the relevant guide.


Core Architecture

All Python Databricks apps follow this pattern:

app-directory/
├── app.py                 # Main application (or framework-specific name)
├── models.py              # Pydantic data models
├── backend.py             # Data access layer
├── requirements.txt       # Additional Python dependencies
├── app.yaml               # Databricks Apps configuration
└── README.md

Backend Toggle Pattern

import os
from databricks.sdk.core import Config

USE_MOCK = os.getenv("USE_MOCK_BACKEND", "true").lower() == "true"

if USE_MOCK:
    from backend_mock import MockBackend as Backend
else:
    from backend_real import RealBackend as Backend

backend = Backend()

SQL Warehouse Connection (shared across all frameworks)

from databricks.sdk.core import Config
from databricks import sql

cfg = Config()  # Auto-detects credentials from environment
conn = sql.connect(
    server_hostname=cfg.host,
    http_path=f"/sql/1.0/warehouses/{os.getenv('DATABRICKS_WAREHOUSE_ID')}",
    credentials_provider=lambda: cfg.authenticate,
)

Pydantic Models

from pydantic import BaseModel, Field
from datetime import datetime
from enum import Enum

class Status(str, Enum):
    ACTIVE = "active"
    PENDING = "pending"

class EntityOut(BaseModel):
    id: str
    name: str
    status: Status
    created_at: datetime

class EntityIn(BaseModel):
    name: str = Field(..., min_length=1)
    status: Status = Status.PENDING

Common Issues

IssueSolution
Connection exhaustedUse @st.cache_resource (Streamlit) or connection pooling
Auth token not foundCheck x-forwarded-access-token header — only available when deployed, not locally
App won't startCheck app.yaml command matches framework; check databricks apps logs <name>
Resource not accessibleAdd resource via UI, verify SP has permissions, use valueFrom in app.yaml
Import error on deployAdd missing packages to requirements.txt (pre-installed packages don't need listing)
Lakebase app crashes on startpsycopg2/asyncpg are NOT pre-installed — MUST add to requirements.txt
Port conflictApps must bind to DATABRICKS_APP_PORT env var (defaults to 8000). Never use 8080. Streamlit is auto-configured; for others, read the env var in code or use 8000 in app.yaml command
Streamlit: set_page_config errorst.set_page_config() must be the first Streamlit command
Dash: unstyled layoutAdd dash-bootstrap-components; use dbc.themes.BOOTSTRAP
Slow queriesUse Lakebase for transactional/low-latency; SQL warehouse for analytical queries

Platform Constraints

ConstraintDetails
RuntimePython 3.11, Ubuntu 22.04 LTS
Compute2 vCPUs, 6 GB memory (default)
Pre-installed frameworksDash, Streamlit, Gradio, Flask, FastAPI, Shiny
Custom packagesAdd to requirements.txt in app root
NetworkApps can reach Databricks APIs; external access depends on workspace config
User authGA, on by default — no admin enablement required; declare user_api_scopes to add scopes

Official Documentation

Related Skills

  • databricks-apps — the default for new Databricks Apps (AppKit / Node / TypeScript + React); load it first unless a Python backend is explicitly required
  • databricks-python-sdk — WorkspaceClient, OpenAI-compatible foundation-model client, model-serving invocation; the same patterns work inside a Databricks App
  • databricks-vector-search — AI Search (Vector Search) indexes and high-QPS queries with AISearchClient
  • databricks-lakebase — persistent PostgreSQL state (autoscaling managed PG with branching)
  • databricks-model-serving — endpoint lifecycle for ML models an App calls
  • databricks-dabs — deploying apps via DABs

Databricks Apps Development

FIRST: Use the parent databricks-core skill for CLI basics, authentication, and profile selection.

Package manager — resolve this before any install or command below:

Databricks Apps officially supports npm and pnpm only.

  1. Detect the project's manager, in priority order: project instructions → package.json's packageManager field → lockfile (pnpm-lock.yaml → pnpm, package-lock.json → npm).
  2. If signals conflict, or if an unsupported manager (yarn, bun, etc.) is detected, stop and ask the user which manager to use before proceeding.
  3. Only if none is established, default to pnpm.

Throughout this skill and its guides, substitute what you resolved:

  • <pm> → that manager, for installs and package scripts (<pm> install, <pm> run typegen).

<pm> is a documentation placeholder, not a shell variable — expand it, don't type it.

Is this even a Databricks App? Don't assume an app is the only way to show data. For a simple "dashboard with a few charts" and no app-specific need, the managed AI/BI (Lakeview) dashboard is the simpler path → use the databricks-aibi-dashboards skill, not this one. Reach for a custom Databricks App only when the user needs something AI/BI can't give them — bespoke interactivity/components, write-back, embedded or auth-gated workflows, a Genie/chat surface inside the app — or explicitly asks for an app. If it's genuinely ambiguous, surface both options (managed AI/BI dashboard vs custom app) and let the user choose instead of defaulting to an app. Once an app is the right call, it's fine for this skill to favor an app-based dashboard, and databricks-app-design covers building it well.

For data UI design (required for any data-displaying app): once you've confirmed the user wants a custom-code app (not a managed AI/BI dashboard — see above), if the app shows ANY data — a KPI/overview page, report, chart, table, query results, OR a conversational / chat / Genie natural-language assistant — you MUST use the databricks-app-design skill (alongside this one) to decide layout, charts, KPIs, semantic color, required states, and AI-result trust, and map them to AppKit components. This includes chat/Genie apps, not just static data views — if in doubt, use it.

Build apps that deploy to Databricks Apps platform.

Required Reading by Phase

PhaseREAD BEFORE proceeding
Scaffolding⚠️ STOP — review the State Storage Guidance and complete the Data Access Decision Gate below before scaffolding. Parent databricks-core skill (auth, warehouse discovery); then run databricks apps manifest + databricks apps init with --features and --set (see AppKit section below)
Writing SQL queriesSQL Queries Guide
Writing UI componentsFrontend Guide
Using useAnalyticsQueryAppKit SDK
Querying a governed UC Metric ViewMetric Views Guide
Adding API endpointsCustom Endpoints Guide
Using Lakebase (OLTP database)Lakebase Guide
Adding Genie chat / Genie-powered appsGenie Guide — follow the Genie agent workflow below
Using Model Serving (ML inference)Model Serving Guide
Generating or editing images (text-to-image, image-to-image)Image Generation Guide
Hosting an AI agent (tool-using chatbot, beta)Agents Guide — import from @databricks/appkit/beta
Typed data contracts (proto-first design)Proto-First Guide and Plugin Contracts
Managing files in UC VolumesFiles Guide
Triggering / monitoring Lakeflow Jobs from the appJobs Guide
Platform rules (permissions, deployment, limits)Platform Guide — READ for ALL apps including AppKit
Non-AppKit app (Streamlit, FastAPI, Flask, Gradio, Next.js, etc.)Other Frameworks

Generic Guidelines

  • App name: Platform App resource names allow up to 30 lowercase letters, numbers, and hyphens (no underscores). For AppKit projects created with databricks apps init, keep the base name to 26 characters because its development workflow adds a dev- prefix. DABs does not add that prefix.
  • Validation: databricks apps validate --profile <PROFILE> before deploying.
  • Smoke tests (AppKit only): ALWAYS update tests/smoke.spec.ts selectors BEFORE running validation. Default template checks for "Minimal Databricks App" heading and "hello world" text — these WILL fail in your custom app. See testing guide (selector APIs, the 1 MB result-size cap).
  • AppKit version: never override the @databricks/appkit or @databricks/appkit-ui version in package.json — databricks apps init sets the correct version. Do not install a different version of either package unless explicitly asked by the user. If you need a different version, re-scaffold with databricks apps init --version <version>.
  • Authentication: covered by parent databricks-core skill.
  • AppKit API surface: before writing code that calls AppKit APIs (createApp, plugin shapes, useAnalyticsQuery, useMetricView, etc.), run npx @databricks/appkit docs <section> and use the actual signature. Training data has stale shapes; a single invented signature fails tsc --noEmit during validate. The docs ship with the installed AppKit and are the authoritative source.
  • TypeScript casts: never use as unknown as <T> double-assertions — appkit lint enforces no-double-type-assertion and one violation fails the entire validate step. Instead: narrow with Zod (z.infer<typeof schema>), use a runtime type guard, or write a typed mapper function. If a query result needs reshaping, type the row schema via queryKey types rather than casting.

Project Structure (after databricks apps init --features analytics)

  • client/src/App.tsx — main React component (start here)
  • config/queries/*.sql — SQL query files (queryKey = filename without .sql)
  • config/metric-views/definitions.json — governed UC Metric View bindings (optional; see Metric Views)
  • server/server.ts — backend entry (onPluginsReady + Express routes)
  • tests/smoke.spec.ts — smoke test (⚠️ MUST UPDATE selectors for your app)
  • client/src/appKitTypes.d.ts — auto-generated types (<pm> run typegen)

Project Structure (after databricks apps init --features lakebase)

  • server/server.ts — backend with Lakebase pool + Express routes
  • client/src/App.tsx — React frontend
  • app.yaml — manifest with database resource declaration
  • package.json — includes @databricks/lakebase dependency
  • Note: No config/queries/ — Lakebase apps use appkit.lakebase.query() in Express routes, not SQL files

Data Discovery

Before writing any SQL, use the parent databricks-core skill for data exploration — search information_schema by keyword, then batch discover-schema for the tables you need. Do NOT skip this step.

State Storage Guidance (evaluate BEFORE the Decision Gate):

If the user's app description involves storing or persisting data — forms, CRUD operations, user submissions, orders, todos, or other user-generated content — the app likely needs a Lakebase database.

  1. Ask the user whether the app needs persistent storage (Lakebase) before scaffolding. Do not silently add Lakebase.
  2. If confirmed, ask whether to reuse an existing Lakebase project or create a new one (same as the Genie flow below) — never create one silently. Use the databricks-lakebase skill to obtain the branch and database resource names:
    • Reusing: list with databricks postgres list-projects, then list-branches / list-databases, and let the user pick the project, branch, and database; confirm which schema the app will own (a fresh/dedicated schema avoids the service-principal ownership conflict).
    • Creating: create a new project (it auto-provisions a production branch + databricks_postgres database).
  3. Scaffold with --features lakebase and pass --set lakebase.postgres.branch=<BRANCH_NAME> --set lakebase.postgres.database=<DATABASE_NAME>.
  4. If the app also reads from Unity Catalog tables, proceed to the Data Access Decision Gate below to determine whether to add --features analytics or use Lakebase synced tables.

Do NOT add Lakebase to analytics, dashboard, or visualization apps unless the user explicitly requests persistent write-back storage. Read-only data display, filters, and preferences do not require a database.

Development Workflow (FOLLOW THIS ORDER)

Data Access Decision Gate (REQUIRED before scaffolding):

If the app reads from Unity Catalog / lakehouse tables, you MUST show the comparison below to the user and ask them to choose. Do not skip this. Do not choose for them.

(A) Lakebase synced tables(B) Analytics
SpeedSub-second responsesTakes a few seconds
Best forFull-text search, typeahead, autocomplete, real-time lookups, operational appsDashboards, charts, aggregations, KPIs, filtered queries, browsing
How it worksData synced from Delta into Lakebase PostgresQueries run on SQL warehouse at read time

After showing the table, add a brief recommendation. Default to recommending Analytics (B) for most read-only apps — dashboards, charts, filtered queries, browsing, and aggregations. Recommend Lakebase synced tables (A) only when the app needs sub-second latency for full-text search, typeahead/autocomplete, real-time lookups by ID, or operational data serving. Note: "search" or "filter" in a prompt usually means SQL WHERE clauses (Analytics), not full-text search (Lakebase). Always let the user make the final call.

After the user chooses:

  • (A) Lakebase synced tables → scaffold with --features lakebase. See Lakebase Guide for full workflow.
  • (B) Analytics → scaffold with --features analytics. Within Analytics, if the data is a governed UC Metric View (pre-defined semantic measures/dimensions), query it via the metric-view path — see Metric Views — otherwise write config/queries/ SQL.
  • Both → scaffold with --features analytics,lakebase if the app needs both patterns.
  • If the app does NOT read Unity Catalog data (pure CRUD, Genie, Model Serving), skip this gate and scaffold with the appropriate --features flag.

Analytics apps (--features analytics):

  1. Create SQL files in config/queries/
  2. Run <pm> run typegen — verify all queries show ✓
  3. Read client/src/appKitTypes.d.ts to see generated types
  4. THEN write App.tsx using the generated types
  5. Update tests/smoke.spec.ts selectors
  6. Run databricks apps validate --profile <PROFILE>

DO NOT write UI code before running typegen — types won't exist and you'll waste time on compilation errors.

Lakebase apps (--features lakebase): No SQL files or typegen. See Lakebase Guide for the onPluginsReady pattern: initialize schema at startup, register Express routes in server/server.ts, then build the React frontend.

When to Use What

After completing the decision gate above, use this routing table:

  • Read analytics data → display in chart/table: Use visualization components with queryKey prop
  • Read analytics data → custom display (KPIs, cards): Use useAnalyticsQuery hook
  • Read analytics data → need computation before display: Still use useAnalyticsQuery, transform client-side
  • Read a governed UC Metric View (semantic layer): Use the useMetricView hook + config/metric-views/definitions.json — see Metric Views
  • Read lakehouse data at low latency (lookups, search, catalogs): Use Lakebase synced tables — see Lakebase Guide
  • Read/write persistent data (users, orders, CRUD state): Use Lakebase via Express routes in onPluginsReady — see Lakebase Guide
  • Natural language query interface over tables (Genie): Use genie() plugin — see Genie Guide
  • Call ML model endpoint: Use serving() plugin — see Model Serving Guide
  • Call a foundation model / UC model service (system.ai.*): Declare a uc_securable (MODEL_SERVICE) resource and call the service name on /ai-gateway/mlflow/v1 — see Unity Catalog model services and the databricks-unity-gateway skill (not serving(), which takes serving-endpoint names)
  • Generate or edit images from a prompt: streaming Express route (or an agents tool for chat UIs) + native fetch to an image-capable Foundation Model — see Image Generation Guide
  • AI agent that calls tools (SQL, files, Genie, MCP), delegates to sub-agents, or streams a multi-turn chat with human-in-the-loop approval: Use the agents() plugin (import from @databricks/appkit/beta, beta) — see Agents Guide
  • Trigger or monitor a Lakeflow Job from the app: Use the jobs() plugin — see Jobs Guide
  • ⚠️ NEVER add custom endpoints to run SELECT queries against the warehouse — always use SQL files in config/queries/
  • ⚠️ NEVER use useAnalyticsQuery for Lakebase data — it queries the SQL warehouse only

Frameworks

AppKit (Recommended)

TypeScript/React framework with type-safe SQL queries and built-in components.

Official Documentation — the source of truth for all API details. Use the project-local appkit CLI from the installed @databricks/appkit package:

npx @databricks/appkit docs                            # ← ALWAYS start here to see available pages
npx @databricks/appkit docs <query>                    # view a section by name or doc path
npx @databricks/appkit docs --full                     # full index with all API entries
npx @databricks/appkit docs "appkit-ui API reference"   # example: section by name
npx @databricks/appkit docs ./docs/plugins/analytics.md  # example: specific doc file

DO NOT guess doc paths. Run without args first, pick from the index. The <query> argument accepts both section names (from the index) and file paths. Docs are the authority on component props, hook signatures, and server APIs — skill files only cover anti-patterns and gotchas.

App Manifest and Scaffolding

Agent workflow for scaffolding: get the manifest first, then build the init command.

  1. Get the manifest (JSON schema describing plugins and their resources):

    databricks apps manifest --profile <PROFILE>
    # See plugins available in a specific AppKit version:
    databricks apps manifest --version <VERSION> --profile <PROFILE>
    # Custom template:
    databricks apps manifest --template <GIT_URL> --profile <PROFILE>
    

    The output defines:

    • Plugins: each has a key (plugin ID for --features), plus requiredByTemplate, and resources.
    • requiredByTemplate: If true, that plugin is mandatory for this template — do not add it to --features (it is included automatically); you must still supply all of its required resources via --set. If false or absent, the plugin is optional — add it to --features only when the user's prompt indicates they want that capability (e.g. analytics/SQL), and then supply its required resources via --set.
    • Resources: Each plugin has resources.required and resources.optional (arrays). Each item has resourceKey and fields (object: field name → description/env). Use --set <plugin>.<resourceKey>.<field>=<value> for each required resource field of every plugin you include.
  2. Scaffold (use the Databricks CLI only):

    databricks apps init --name <NAME> --features <plugin1>,<plugin2> \
      --set <plugin1>.<resourceKey>.<field>=<value> \
      --set <plugin2>.<resourceKey>.<field>=<value> \
      --description "<DESC>" --run none --profile <PROFILE>
    # --run none: skip auto-run after scaffolding (review code first)
    # With custom template:
    databricks apps init --template <GIT_URL> --name <NAME> --features ... --set ... --profile <PROFILE>
    

    Optionally use --version <VERSION> to target a specific AppKit version.

    • Required: --name, --profile. Name: ≤26 chars, lowercase letters/numbers/hyphens only. Use --features only for optional plugins the user wants (plugins with requiredByTemplate: false or absent); mandatory plugins must not be listed in --features.
    • Resources: Pass --set for every required resource (each field in resources.required) for (1) all plugins with requiredByTemplate: true, and (2) any optional plugins you added to --features. Add --set for resources.optional only when the user requests them.
    • Discovery: Use the parent databricks-core skill to resolve IDs (e.g. warehouse: databricks warehouses list --profile <PROFILE> or databricks experimental aitools tools get-default-warehouse --profile <PROFILE>).

DO NOT guess plugin names, resource keys, or property names — always derive them from databricks apps manifest output. Example: if the manifest shows plugin analytics with a required resource resourceKey: "sql-warehouse" and fields: { "id": ... }, include --set analytics.sql-warehouse.id=<ID>.

Execution identity (auth mode): --auth-mode obo|sp (app default) and per-resource --set <plugin>.<resourceKey>.authMode=obo|sp|both need databricks CLI >= v1.20.0; omit them on older CLIs (everything binds to the service principal). Default to SP; choose OBO only when the app must enforce the viewing user's own permissions (row-level or per-user data access). Flag rules, errors, and what each mode emits: Platform Guide.

Scaffolding Rules Protocol: if databricks apps manifest emits scaffolding.rules (template-level and per plugin, as must / should / never), they override this skill. Run Before init rules before databricks apps init and After init rules after it; if a plugin must contradicts a template never on the same target, STOP and ask the user. Gather, precedence, and reporting steps: AppKit Overview § Scaffolding Rules Protocol.

READ AppKit Overview for project structure, workflow, and pre-implementation checklist.

Genie Agent Workflow: for a Genie-powered app, ask which UC tables to query and whether to reuse or create a space; don't start by asking for a Genie Space ID. Steps: Genie Guide § Agent workflow.

Read the Genie Guide for configuration, SSE endpoints, and frontend integration.

Common Scaffolding Mistakes

# ❌ WRONG: name is NOT a positional argument
databricks apps init --features analytics my-app-name
# → "unknown command" error

# ✅ CORRECT: use --name flag
databricks apps init --name my-app-name --features analytics --set "..." --profile <PROFILE>

Directory Naming

databricks apps init creates directories in kebab-case matching the app name. App names must be lowercase with hyphens only (≤26 chars).

Other Frameworks (Streamlit, FastAPI, Flask, Gradio, Dash, Next.js, etc.)

Databricks Apps supports any framework that runs as an HTTP server. LLMs already know these frameworks — the challenge is Databricks platform integration.

READ Other Frameworks Guide BEFORE building any non-AppKit app. It covers port/host configuration, app.yaml and databricks.yml setup, dependency management, networking, and framework-specific gotchas.

Post-Deploy Verification

After deploying, verify the app is running:

databricks apps get <app-name> --profile <PROFILE> -o json   # Check app_status.state: RUNNING; the `url` field is the app's URL
databricks apps logs <app-name> --follow --profile <PROFILE>  # Stream live logs (Ctrl+C to stop)

Note: databricks apps logs requires OAuth authentication and does not work with PAT. Use databricks apps get for status checks if using PAT auth.

Databricks

Core skill for Databricks CLI, authentication, and data exploration.

Product Skills

For specific products, use dedicated skills:

  • databricks-jobs - Lakeflow Jobs development and deployment
  • databricks-pipelines - Lakeflow Spark Declarative Pipelines (batch and streaming data pipelines)
  • databricks-serverless - Compute choice for jobs and pipelines (serverless, performance mode, classic fallback)
  • databricks-apps - Full-stack TypeScript app development and deployment
  • databricks-lakebase - Lakebase Postgres Autoscaling project management
  • databricks-model-serving - Model Serving endpoint management and inference

For data discovery, exploration, and query generation — finding tables, answering natural-language questions about the data, or generating SQL — use databricks-data-discovery (it asks Genie One first, then falls back to manual exploration). If it isn't installed, use the AI-tool commands below and Manual Data Exploration.

Prerequisites

  1. CLI installed and current: the CLI must be >= v1.0.0. A CLI that is present but too old is not "good enough" — it must be upgraded, not worked around.

    • Run the floor check in CLI Installation. If it reports UPGRADE or INSTALL: STOP and follow that reference file to upgrade or install — do not proceed or tell the user their CLI is fine.
    • If another skill routed you here to upgrade (it needs a newer CLI than v1.0.0 — e.g. databricks-setup-local needs v1.16.0), that skill already detected the gap. A passing v1.0.0 floor check is not enough: follow the Update / repair procedures to install the latest stable, which satisfies any skill's floor.
    • Note: In sandboxed environments (Cursor IDE, containers), install commands write outside the workspace and may be blocked. Present the install command to the user and ask them to run it in their own terminal.
    • Exception: If CLI installation is blocked (sandboxed containers, restricted environments), ask the user whether to fall back to direct REST API calls using DATABRICKS_HOST and DATABRICKS_TOKEN environment variables if present in the shell. See the Databricks REST API docs.
  2. Authenticated: databricks auth profiles

Profile Selection - CRITICAL

NEVER auto-select a profile.

  1. List profiles: databricks auth profiles
  2. Present ALL profiles to user with workspace URLs
  3. Let user choose (even if only one exists)
  4. Offer to create new profile if needed

Claude Code - IMPORTANT

Each Bash command runs in a separate shell session.

# WORKS: --profile flag
databricks apps list --profile my-workspace

# WORKS: chained with &&
export DATABRICKS_CONFIG_PROFILE=my-workspace && databricks apps list

# DOES NOT WORK: separate commands
export DATABRICKS_CONFIG_PROFILE=my-workspace
databricks apps list  # profile not set!

Tag CLI Commands With Loaded Skills

When databricks --version is v1.27.0 or later, add --skills-invoked <skills> to every databricks CLI command you run, where <skills> is a comma-separated list of the bare names of the Databricks skills loaded for this task, for example databricks jobs list --skills-invoked databricks-core,databricks-jobs. Put it right after the subcommand, before any -- separator, because arguments after -- go to another program (for example databricks psql --skills-invoked databricks-core,databricks-lakebase --project <ID> -- -c "..."). Use bare skill names only, with no plugin: prefix. Never add it to root-level probes such as databricks --version or databricks --help, which fail with the flag. Leave it off on older CLIs, which reject the flag. Do not add it to scripts or code written for the user.

Data Exploration — Use AI Tools

Use these instead of manually navigating catalogs/schemas/tables:

# discover table structure (columns, types, sample data, stats)
databricks experimental aitools tools discover-schema catalog.schema.table --profile <PROFILE>

# run ad-hoc SQL queries
databricks experimental aitools tools query "SELECT * FROM table LIMIT 10" --profile <PROFILE>

# find the default warehouse
databricks experimental aitools tools get-default-warehouse --profile <PROFILE>

Names are literal. Use catalog/schema/table names exactly as given — never change a hyphen to an underscore or otherwise normalize them. In SQL, backtick-quote any name part with special characters (e.g. `my-catalog`.schema.table); unquoted hyphens cause a parse error.

These commands are first-class for running known SQL and profiling — Genie isn't required for that. For natural-language data questions, locating data you can't pin down, or generating a query from a question, prefer the databricks-data-discovery skill (above) if it's installed. See Manual Data Exploration for the full command surface, quoting rules, and troubleshooting.

Quick Reference

⚠️ CRITICAL: Some commands use positional arguments, not flags

# current user
databricks current-user me --profile <PROFILE>

# list resources
databricks apps list --profile <PROFILE>
databricks jobs list --profile <PROFILE>
databricks clusters list --profile <PROFILE>
databricks warehouses list --profile <PROFILE>
databricks pipelines list --profile <PROFILE>
databricks serving-endpoints list --profile <PROFILE>

# ⚠️ Unity Catalog — POSITIONAL arguments (NOT flags!)
databricks catalogs list --profile <PROFILE>

# ✅ CORRECT: positional args
databricks schemas list <CATALOG> --profile <PROFILE>
databricks tables list <CATALOG> <SCHEMA> --profile <PROFILE>
databricks tables get <CATALOG>.<SCHEMA>.<TABLE> --profile <PROFILE>

# ❌ WRONG: these flags/commands DON'T EXIST
# databricks schemas list --catalog-name <CATALOG>    ← WILL FAIL
# databricks tables list --catalog <CATALOG>           ← WILL FAIL
# databricks sql-warehouses list                       ← doesn't exist, use `warehouses list`
# databricks execute-statement                         ← doesn't exist, use `experimental aitools tools query`
# databricks sql execute                               ← doesn't exist, use `experimental aitools tools query`

# When in doubt, check help:
# databricks schemas list --help

# get details
databricks apps get <NAME> --profile <PROFILE>
databricks jobs get --job-id <ID> --profile <PROFILE>
databricks clusters get --cluster-id <ID> --profile <PROFILE>

# bundles
databricks bundle init --profile <PROFILE>
databricks bundle validate --profile <PROFILE>
databricks bundle deploy -t <TARGET> --profile <PROFILE>
databricks bundle run <RESOURCE> -t <TARGET> --profile <PROFILE>

Troubleshooting

ErrorSolution
cannot configure default credentialsUse --profile flag or authenticate first
configuration does not support OAuth tokensThe command requires OAuth (e.g., databricks apps logs). Re-authenticate with databricks auth login --host <URL> --profile <PROFILE>. See CLI Authentication.
PERMISSION_DENIEDCheck workspace/UC permissions
RESOURCE_DOES_NOT_EXISTVerify resource name/id and profile

Required Reading by Task

TaskREAD BEFORE proceeding
First time setupCLI Installation
Auth issues / new workspaceCLI Authentication
Exploring tables/schemasManual Data Exploration (or databricks-data-discovery if installed)
Deploying jobs/pipelinesUse /databricks-dabs

Reference Guides

Declarative Automation Bundles (DABs)

Use this skill for any bundle-related request including creating, configuring, validating, deploying, running, and managing Databricks resources through DABs.

Reference Documentation

The following reference files provide detailed guidance for specific bundle tasks:

  • Bundle Structure - Bundle structure, databricks.yml configuration, resource definitions, path resolution, variables, and multi-environment targets
  • SDP Pipelines - Spark Declarative Pipeline configurations for DABs
  • SQL Alerts - SQL Alert schemas and configuration (critical - API differs from other resources)
  • Deploy and Run - Validation, deployment, running resources, monitoring logs, and troubleshooting common issues
  • Resource Permissions - Permission levels and access control for bundle resources, per-resource-type levels, grants vs permissions

When to Use This Skill

Load this skill for any request involving:

  • Creating new bundle projects or resources
  • Configuring databricks.yml or resource YAML files
  • Setting up multi-environment deployments (dev/prod targets)
  • Deploying or running bundle resources
  • Managing permissions for bundle resources
  • Troubleshooting bundle validation or deployment errors
  • Working with specific resource types (dashboards, jobs, pipelines, alerts, volumes, apps)

General Guidelines

  1. Always validate after configuration changes - Use bundle validate --strict --target <target> after any change
  2. Use reference documentation - Consult the appropriate reference file for detailed patterns and examples
  3. Follow naming conventions - Resource files should use <name>.<resource_type>.yml format
  4. Path resolution is critical - Paths differ based on file location (see Bundle Structure reference)
  5. Preserve existing structure - Keep user comments and structure when editing YAML files
  6. Use variables - Parameterize catalog, schema, and warehouse for multi-environment support
  7. Namespace App names - App names are workspace-global and limited to 30 characters. With Databricks CLI 0.270.0 or later, shared-development defaults should include the app and ${workspace.current_user.domain_friendly_name}; override production with a stable name, and persist an explicit local value when the default is invalid, collides, or must distinguish multiple non-production targets in one workspace. On older CLI versions, require an explicit app_name value instead

Required App Completion Contract

Before validating any bundle that creates or changes an App, re-read the final bundle YAML and confirm all of the following structural requirements:

  • Each App resource uses a dedicated name variable, such as name: ${var.app_name}, while preserving an existing equivalent variable when present.
  • For Databricks CLI 0.270.0 or later, each App name variable defaults to a recognizable App prefix plus ${workspace.current_user.domain_friendly_name} for development, and the production target overrides it with a stable name. On older CLI versions, App name variables have no default and each developer must provide values.
  • No development App name uses ${workspace.current_user.short_name}, a bare ${bundle.target}, or a hardcoded value.

Only after this check, run databricks bundle validate --strict --target <target> --output json and inspect each resolved resources.apps.<key>.name. Each name must contain only lowercase letters, digits, and hyphens and be at most 30 characters. If a resolved name is invalid, collides after normalization, or multiple non-production targets share a workspace, persist a shorter, distinct value in the uncommitted .databricks/bundle/<target>/variable-overrides.json file and validate again. Successful validation alone does not satisfy this contract because validation does not catch every invalid or non-namespaced App name.

Documentation

Databricks Data Discovery

This skill routes data work — decide first:

  • the request is about the data — finding it, asking questions of it, or generating SQL → delegate to Genie One: databricks genie ask -s <session-label> "..." (see Routing below).
  • writing files or anything else → use your own coding-agent tools.

Genie One just needs an authenticated CLI profile (the parent databricks-core skill covers auth/profiles if you need it) — but route the data request to Genie first; don't detour into manual catalog browsing.

Routing — Genie vs your coding agent

Route to Genie when the request is about the data:

  • Finding or locating data — "what tables are in wanderbricks?", "where does X live?", which catalog/schema holds something. Let Genie resolve a fuzzy or partial name across all your data instead of guessing or grepping catalogs yourself.
  • Questions about the data — "how many…", "what's the average…", "which … the most…", trends, breakdowns.
  • Generating SQL from a question — a first-cut query you can run, adapt, or drop into a file.
  • Understanding or profiling data to inform a query, dashboard, or app.

Route to your own coding agent (do NOT use Genie) for everything else:

  • Writing or editing files — a .sql file, notebook, dashboard, app, config. Genie finds the data and produces the SQL; you write the file.
  • Authoring jobs, pipelines, apps, or DABs — use those product skills.
  • DDL or writes — CREATE / INSERT / UPDATE / DELETE.
  • General programming, git, infra, and any non-data task.

Key principle: data discovery, data questions, and query generation → Genie One. Everything else → your coding agent.

Why prefer Genie for data questions

Genie runs inside the Databricks data plane with governed, first-hand access to the org's Unity Catalog metadata, metric views, and curated semantic context — context you do not have when reverse-engineering schemas with ad-hoc SQL. For data questions it is often higher-quality and more performant than doing the discovery yourself, and it keeps improving as a managed Databricks capability. Don't default to writing your own discovery SQL just because you can.

How to ask Genie

Always pass a session label with -s, and prefer reusing the same one: a follow-up can only continue a conversation if the first ask set the session label, and reusing it lets later questions build on everything asked so far ("summarize all of the above"). Use a fresh session label only to start a deliberately separate session, or distinct session labels to run several in parallel.

The command is databricks genie ask (CLI >= v1.9.0). On an older CLI it lives under databricks experimental genie ask — same flags and behavior; use that exact fallback if databricks genie ask is not found.

# Always pass a session label, and reuse the SAME one so follow-ups build on each other
databricks genie ask -s trips "How many bookings were there last week?"
databricks genie ask -s trips "Break that down by destination"
databricks genie ask -s trips "Summarize all of the above"

# --include-sql also prints the SQL Genie ran (use it to generate a query, too)
databricks genie ask -s trips "Write SQL for the top 5 destinations by revenue" --include-sql

# --output json gives a parseable result
databricks genie ask -s trips "Top 5 destinations by revenue" --output json
# → {"status":"completed","conversation_id":"…","text":"…","tool_calls":[{"name":"execute_sql","sql":"…","title":"…"}]}

# Older CLI (< v1.9.0) — same command under the deprecated experimental alias:
#   databricks experimental genie ask -s trips "How many bookings were there last week?"

Genie searches across all the data you can see, runs SQL, and streams a grounded answer — rendered with the executed SQL and, where it helps, a terminal chart. It auto-resolves a SQL warehouse (override with --warehouse-id); nothing to pick or set up.

  • Streams live: the answer, the agent's steps, and any SQL/results appear as they arrive. Answers usually take ~5–30s; a stalled stream (no data for ~10 min) fails with a clear message, and Ctrl-C or kill (SIGTERM) cancels cleanly.
  • Picking a session label: any string works — a topic like trips, or $$ for a per-shell session. Default to reusing one session label so follow-ups keep full context; use a fresh one only for a deliberately separate session. An expired session label just starts fresh on the next ask. No id to copy around.
  • Parallelism: to run sessions at the same time, give each its own session label (-s q1, -s q2, …) — independent session labels don't interfere. Within a single session label keep calls sequential: send a follow-up after the previous turn returns, and never fire two asks at once on the same session label (they'd split into two conversations and only one mapping would survive).
  • Structured output: --output json gives {status, conversation_id, text, tool_calls[]}, where tool_calls includes the SQL Genie executed; --raw dumps the raw event stream. Note --output json buffers and prints once at the end (no live streaming) — use it for parsing, the default text output for interactive use.
  • Generating a query: ask Genie to "write SQL for …" and read the SQL from the response (it's in the answer text, and --include-sql also shows the query Genie ran to verify it — so the SQL is known-good). Genie resolves the schema and joins for you, so this beats hand-writing SQL against unfamiliar tables.
  • For exact/full rows: Genie shows a preview inline; to pull the complete result set locally, copy its SQL (--include-sql or the JSON tool_calls) into the parent's ... aitools tools query "<SQL>".
  • A non-answer is a message, not an error: if Genie refuses or "couldn't find relevant data," don't retry — use the manual fallback below.

Naming: "Genie One" is the current name for this cross-data chat — formerly "Databricks One", then "OneChat" (the backend tool is still literally named onechat). All the same thing.

If Genie One isn't available — manual fallback

Only fall back if Genie One is genuinely unavailable — first verify with databricks genie ask --help (or databricks experimental genie ask --help on a CLI older than v1.9.0); don't assume the command is missing. When Genie One isn't enabled, the CLI is too old to have either form of genie ask, or Genie can't cover the question, do the discovery yourself with the parent skill's commands — see Manual Data Exploration (keyword search via information_schema, discover-schema, and tools query). Running known SQL or profiling a known table that way is perfectly fine on its own. Do not default to databricks tables list or raw UC REST for data-location questions — invoke this skill and ask Genie first.

Relationship to the Genie One MCP

Databricks also offers this capability as a managed MCP server — the Genie One MCP. This skill delivers the same functionality through the Databricks CLI, with no MCP server to configure or host. More broadly, the Databricks Agent Skills cover the same ground as Databricks' managed MCP servers, so you don't need any MCP wired up to use this. If you already run the Genie One MCP, use whichever you prefer — they hit the same Genie backend.

Related Skills

  • databricks-genie-agents — build and manage Genie Agents: curated agents that let you or a group ask questions of specific data (create, configure, import/export).
  • databricks-core (parent) — CLI auth, profiles, and the manual data exploration reference used as the fallback.

Databricks SQL (DBSQL) - Advanced Features

Quick Reference

FeatureKey SyntaxSinceReference
SQL ScriptingBEGIN...END, DECLARE, IF/WHILE/FORDBR 16.3+references/sql-scripting.md
Stored ProceduresCREATE PROCEDURE, CALLDBR 17.0+references/sql-scripting.md
Recursive CTEsWITH RECURSIVEDBR 17.0+references/sql-scripting.md
TransactionsBEGIN ATOMIC...ENDGAreferences/sql-scripting.md
Materialized ViewsCREATE MATERIALIZED VIEWPro/Serverlessreferences/materialized-views-pipes.md
Temp TablesCREATE TEMPORARY TABLEAllreferences/materialized-views-pipes.md
Pipe Syntax|> operatorDBR 16.1+references/materialized-views-pipes.md
Geospatial (H3)h3_longlatash3(), h3_polyfillash3()DBR 11.2+references/geospatial-collations.md
Geospatial (ST)ST_Point(), ST_Contains(), 80+ funcsDBR 16.0+references/geospatial-collations.md
CollationsCOLLATE, UTF8_LCASE, locale-awareDBR 16.1+references/geospatial-collations.md
AI Functionsai_query(), ai_classify(), 11+ funcsDBR 15.1+references/ai-functions.md
http_requesthttp_request(conn, ...)Pro/Serverlessreferences/ai-functions.md
remote_querySELECT * FROM remote_query(...)Pro/Serverlessreferences/ai-functions.md
read_filesSELECT * FROM read_files(...)Allreferences/ai-functions.md
Data ModelingStar schema, Liquid ClusteringAllreferences/best-practices.md

Common Patterns

SQL Scripting - Procedural ETL

BEGIN
  DECLARE v_count INT;
  DECLARE v_status STRING DEFAULT 'pending';

  SET v_count = (SELECT COUNT(*) FROM catalog.schema.raw_orders WHERE status = 'new');

  IF v_count > 0 THEN
    INSERT INTO catalog.schema.processed_orders
    SELECT *, current_timestamp() AS processed_at
    FROM catalog.schema.raw_orders
    WHERE status = 'new';

    SET v_status = 'completed';
  ELSE
    SET v_status = 'skipped';
  END IF;

  SELECT v_status AS result, v_count AS rows_processed;
END

Stored Procedure with Error Handling

CREATE OR REPLACE PROCEDURE catalog.schema.upsert_customers(
  IN p_source STRING,
  OUT p_rows_affected INT
)
LANGUAGE SQL
SQL SECURITY INVOKER
BEGIN
  DECLARE EXIT HANDLER FOR SQLEXCEPTION
  BEGIN
    SET p_rows_affected = -1;
    SIGNAL SQLSTATE '45000'
      SET MESSAGE_TEXT = concat('Upsert failed for source: ', p_source);
  END;

  MERGE INTO catalog.schema.dim_customer AS t
  USING (SELECT * FROM identifier(p_source)) AS s
  ON t.customer_id = s.customer_id
  WHEN MATCHED THEN UPDATE SET *
  WHEN NOT MATCHED THEN INSERT *;

  SET p_rows_affected = (SELECT COUNT(*) FROM identifier(p_source));
END;

-- Invoke:
CALL catalog.schema.upsert_customers('catalog.schema.staging_customers', ?);

Materialized View with Scheduled Refresh

CREATE OR REPLACE MATERIALIZED VIEW catalog.schema.daily_revenue
  CLUSTER BY (order_date)
  SCHEDULE EVERY 1 HOUR
  COMMENT 'Hourly-refreshed daily revenue by region'
AS SELECT
    order_date,
    region,
    SUM(amount) AS total_revenue,
    COUNT(DISTINCT customer_id) AS unique_customers
FROM catalog.schema.fact_orders
JOIN catalog.schema.dim_store USING (store_id)
GROUP BY order_date, region;

Pipe Syntax - Readable Transformations

-- Traditional SQL rewritten with pipe syntax
FROM catalog.schema.fact_orders
  |> WHERE order_date >= current_date() - INTERVAL 30 DAYS
  |> AGGREGATE SUM(amount) AS total, COUNT(*) AS cnt GROUP BY region, product_category
  |> WHERE total > 10000
  |> ORDER BY total DESC
  |> LIMIT 20;

AI Functions - Enrich Data with LLMs

-- Classify support tickets
SELECT
  ticket_id,
  description,
  ai_classify(description, ARRAY('billing', 'technical', 'account', 'feature_request')) AS category,
  ai_analyze_sentiment(description) AS sentiment
FROM catalog.schema.support_tickets
LIMIT 100;

-- Extract entities from text
SELECT
  doc_id,
  ai_extract(content, ARRAY('person_name', 'company', 'dollar_amount')) AS entities
FROM catalog.schema.contracts;

-- General-purpose AI query with structured output
SELECT ai_query(
  'databricks-meta-llama-3-3-70b-instruct',
  concat('Summarize this customer feedback in JSON with keys: topic, sentiment, action_items. Feedback: ', feedback),
  returnType => 'STRUCT<topic STRING, sentiment STRING, action_items ARRAY<STRING>>'
) AS analysis
FROM catalog.schema.customer_feedback
LIMIT 50;

Geospatial - Proximity Search with H3

-- Find stores within 5km of each customer using H3 indexing
WITH customer_h3 AS (
  SELECT *, h3_longlatash3(longitude, latitude, 7) AS h3_cell
  FROM catalog.schema.customers
),
store_h3 AS (
  SELECT *, h3_longlatash3(longitude, latitude, 7) AS h3_cell
  FROM catalog.schema.stores
)
SELECT
  c.customer_id,
  s.store_id,
  ST_Distance(
    ST_Point(c.longitude, c.latitude),
    ST_Point(s.longitude, s.latitude)
  ) AS distance_m
FROM customer_h3 c
JOIN store_h3 s ON h3_ischildof(c.h3_cell, h3_toparent(s.h3_cell, 5))
WHERE ST_Distance(
  ST_Point(c.longitude, c.latitude),
  ST_Point(s.longitude, s.latitude)
) < 5000;

Collation - Case-Insensitive Search

-- Create table with case-insensitive collation
CREATE TABLE catalog.schema.products (
  product_id BIGINT GENERATED ALWAYS AS IDENTITY,
  name STRING COLLATE UTF8_LCASE,
  category STRING COLLATE UTF8_LCASE,
  price DECIMAL(10, 2)
);

-- Queries automatically case-insensitive (no LOWER() needed)
SELECT * FROM catalog.schema.products
WHERE name = 'MacBook Pro';  -- matches 'macbook pro', 'MACBOOK PRO', etc.

http_request - Call External APIs

-- Set up connection first (one-time)
CREATE CONNECTION my_api_conn
  TYPE HTTP
  OPTIONS (host 'https://api.example.com', bearer_token secret('scope', 'token'));

-- Call API from SQL
SELECT
  order_id,
  http_request(
    conn => 'my_api_conn',
    method => 'POST',
    path => '/v1/validate',
    json => to_json(named_struct('order_id', order_id, 'amount', amount))
  ).text AS api_response
FROM catalog.schema.orders
WHERE needs_validation = true;

read_files - Ingest Raw Files

-- Read JSON files from a Volume with schema hints
SELECT *
FROM read_files(
  '/Volumes/catalog/schema/raw/events/',
  format => 'json',
  schemaHints => 'event_id STRING, timestamp TIMESTAMP, payload MAP<STRING, STRING>',
  pathGlobFilter => '*.json',
  recursiveFileLookup => true
);

-- Read CSV with options
SELECT *
FROM read_files(
  '/Volumes/catalog/schema/raw/sales/',
  format => 'csv',
  header => true,
  delimiter => '|',
  dateFormat => 'yyyy-MM-dd',
  schema => 'sale_id INT, sale_date DATE, amount DECIMAL(10,2), store STRING'
);

Recursive CTE - Hierarchy Traversal

WITH RECURSIVE org_chart AS (
  -- Anchor: top-level managers
  SELECT employee_id, name, manager_id, 0 AS depth, ARRAY(name) AS path
  FROM catalog.schema.employees
  WHERE manager_id IS NULL

  UNION ALL

  -- Recursive: direct reports
  SELECT e.employee_id, e.name, e.manager_id, o.depth + 1, array_append(o.path, e.name)
  FROM catalog.schema.employees e
  JOIN org_chart o ON e.manager_id = o.employee_id
  WHERE o.depth < 10  -- safety limit
)
SELECT * FROM org_chart ORDER BY depth, name;

remote_query - Federated Queries

-- Query PostgreSQL via Lakehouse Federation
SELECT *
FROM remote_query(
  'my_postgres_connection',
  database => 'my_database',
  query    => 'SELECT customer_id, email, created_at FROM customers WHERE active = true'
);

Reference Files

Load these for detailed syntax, full parameter lists, and advanced patterns:

FileContentsWhen to Read
references/sql-scripting.mdSQL Scripting, Stored Procedures, Recursive CTEs, TransactionsUser needs procedural SQL, error handling, loops, dynamic SQL
references/materialized-views-pipes.mdMaterialized Views, Temp Tables/Views, Pipe SyntaxUser needs MVs, refresh scheduling, temp objects, pipe operator
references/geospatial-collations.md39 H3 functions, 80+ ST functions, Collation types and hierarchyUser needs spatial analysis, H3 indexing, case/accent handling
references/ai-functions.md13 AI functions, http_request, remote_query, read_files (all options)User needs AI enrichment, API calls, federation, file ingestion
references/best-practices.mdData modeling, performance, Liquid Clustering, anti-patternsUser needs architecture guidance, optimization, or modeling advice

Key Guidelines

  • Always use Serverless SQL warehouses for AI functions, MVs, and http_request
  • Use LIMIT during development with AI functions to control costs
  • Prefer Liquid Clustering over partitioning for new tables (1-4 keys max)
  • Use CLUSTER BY AUTO when unsure about clustering keys
  • Star schema in Gold layer for BI; OBT acceptable in Silver
  • Define PK/FK constraints on dimensional models for query optimization
  • Use COLLATE UTF8_LCASE for user-facing string columns that need case-insensitive search
  • Test SQL via CLI (databricks experimental aitools tools query) or notebooks before deploying. If --warehouse is rejected on your CLI version, set DATABRICKS_WAREHOUSE_ID in the environment instead.

Databricks Documentation Reference

This skill provides access to the complete Databricks documentation index via llms.txt - use it as a reference resource to supplement other skills.

Role of This Skill

This is a reference skill, not an action skill. Use it to:

  • Look up documentation when other skills don't cover a topic
  • Get authoritative guidance on Databricks concepts and APIs
  • Find detailed information to inform CLI commands and SDK usage
  • Discover features and capabilities you may not know about

Always prefer using CLI/SDK for actions and load specific skills for workflows (databricks-python-sdk, databricks-pipelines, etc.). Use this skill when you need reference documentation.

How to Use

Fetch the llms.txt documentation index:

URL: https://docs.databricks.com/llms.txt

Use WebFetch to retrieve this index, then:

  1. Search for relevant sections/links
  2. Fetch specific documentation pages for detailed guidance
  3. Apply what you learn using the appropriate CLI commands or SDK

Documentation Structure

The llms.txt file is organized by category:

  • Overview & Getting Started - Basic concepts and tutorials
  • Data Engineering - Lakeflow, Spark, Delta Lake, pipelines
  • SQL & Analytics - Warehouses, queries, dashboards
  • AI/ML - MLflow, model serving, GenAI
  • Governance - Unity Catalog, permissions, security
  • Developer Tools - SDKs, CLI, APIs, Terraform

Example: Complementing Other Skills

Scenario: User wants to create a Delta Live Tables pipeline

  1. Load databricks-pipelines skill for workflow patterns
  2. Use this skill to fetch docs if you need clarification on specific DLT features
  3. Use databricks pipelines create CLI command to create the pipeline

Scenario: User asks about an unfamiliar Databricks feature

  1. Fetch llms.txt to find relevant documentation
  2. Read the specific docs to understand the feature
  3. Determine which skill/tools apply, then use them

Related Skills

  • databricks-python-sdk - SDK patterns for programmatic Databricks access
  • databricks-pipelines - DLT / Lakeflow pipeline workflows
  • databricks-unity-catalog - Governance and catalog management
  • databricks-model-serving - Serving endpoints and model deployment
  • databricks-mlflow-evaluation - MLflow agent evaluation and scorers; use databricks-mlflow-tracing for tracing and observability

Databricks Execution & Compute

Run code on Databricks. Four execution modes—choose based on workload. Most examples below use the Databricks CLI (Databricks Sandbox is SDK-driven); see the databricks-core skill for install and authentication.

Execution Mode Decision Matrix

AspectDatabricks Connect ⭐Serverless JobInteractive ClusterDatabricks Sandbox
Use forSpark code (ETL, data gen)Heavy processing (ML)State across tool calls, Scala/RUntrusted / agent-generated code, isolated env
StartupInstant~25-50s cold start~5min if stopped~1s (warm start)
StateWithin Python processNoneVia context_idHome dir persisted across stop/restart
LanguagesPython (PySpark)Python, SQLPython, Scala, SQL, RAny (general Linux shell)
DependencieswithDependencies()CLI with environments specInstall on clusterInstall in sandbox, or spec.environment.image_uri (UC container artifact) for a custom image

Decision Flow

Main decision point: if you're using Declarative Automation Bundles (DABs) then follow the instructions of the databricks-dabs skill first. In short, you can use databricks bundle run to run code associated with jobs, pipelines, and other resources. This can be recognized by looking for a databricks.yml file in the project root. If these resources don't exist, or if you're not using DABs, then proceed with the below.

Prefer Databricks Connect for all spark-based workload, then serverless.

Spark-based code? → Databricks Connect (fastest)
  └─ Python 3.12 missing? → Install it + databricks-connect
  └─ Install fails? → Ask user (don't auto-switch modes)

Heavy/long-running (ML)? → Serverless Job (independent)
Need state across calls? → Interactive Cluster (list and ask which one to use)
Scala/R? → Interactive Cluster (list and ask which one to use)
Untrusted / agent-generated code, or need an isolated Linux env? → Databricks Sandbox

How to Run Code

Read the reference file for your chosen mode before proceeding.

Databricks Connect (run locally, prefer when it's pure spark code) → reference

from databricks.connect import DatabricksSession
...
spark = DatabricksSession.builder.profile("my-local-profile").serverless(True).getOrCreate()


python my_spark_script.py

Serverless Job → reference

Pure CLI flow: upload a local file as a workspace notebook, fire a one-time run with databricks jobs submit (create + run in one call, ephemeral — no Jobs UI entry, no retry), then poll + fetch the result. The local file must be a Databricks source notebook — top line # Databricks notebook source (Python) or -- Databricks notebook source (SQL).

1. Upload the local file as a workspace notebook. TARGET_PATH is positional; --file is the local path.

databricks workspace import /Workspace/Users/<user>/.ai_dev_kit/train --file /local/path/to/train.py --format SOURCE --language PYTHON --overwrite

2. Submit the run. Use --no-wait to get {"run_id": N} back immediately; drop it to block until terminated. "client": "4" is required for dependencies to install ("1" silently ignores them).

databricks jobs submit --no-wait --json @submit.json

{
  "run_name": "train-run",
  "tasks": [{
    "task_key": "main",
    "notebook_task": {"notebook_path": "/Workspace/Users/<user>/.ai_dev_kit/train"},
    "environment_key": "ml_env"
  }],
  "environments": [{
    "environment_key": "ml_env",
    "spec": {"client": "4", "dependencies": ["scikit-learn==1.5.2", "mlflow==2.22.0"]}
  }]
}

3. Check state / wait for completion. Life-cycle: PENDING → RUNNING → TERMINATED (or SKIPPED / INTERNAL_ERROR). Only read .state.result_state (SUCCESS / FAILED / CANCELED) once life-cycle is TERMINATED.

databricks jobs get-run <RUN_ID> | jq '{state: .state.life_cycle_state, result: .state.result_state, duration_ms: .execution_duration, url: .run_page_url, task_run_id: .tasks[0].run_id}'

4. Fetch the output / error. Gotcha: get-run-output takes the task run_id (.tasks[0].run_id), NOT the parent run_id from submit. notebook_output.result is the string passed to dbutils.notebook.exit().

databricks jobs get-run-output <TASK_RUN_ID> | jq '{result: .notebook_output.result, error, error_trace}'

Always use dbutils.notebook.exit(<string>) in the notebook — print() is not captured by get-run-output. For JSON results: dbutils.notebook.exit(json.dumps({...})) then parse .notebook_output.result client-side.

Interactive Cluster → reference

Avoid by default — prefer Serverless Job. Only use an interactive cluster when:

  • you have an existing classic cluster already running and available, or
  • you need live, stateful execution across multiple calls (debugging via an execution context), or
  • the user explicitly asks for it.

Interactive clusters are slow to start (3-8 min) and cost money while running. Don't start one implicitly.

Databricks Sandbox → skill

Isolated serverless environment for running untrusted or agent-generated code, driven through the Databricks SDK. Prefer it for agent tool-execution and ephemeral build/test environments rather than Spark workloads. See the databricks-sandbox skill for lifecycle, command execution, and file operations.

CLI Command Map

All compute lifecycle and code-execution actions go through the Databricks CLI. Headline commands:

ActionCommand
Upload local file as workspace notebookdatabricks workspace import <WORKSPACE_PATH> --file <LOCAL> --format SOURCE --language PYTHON --overwrite
Run serverless code (upload + submit + wait)databricks jobs submit --json @submit.json (see Serverless Job section above; with --no-wait for async)
Get run state / waitdatabricks jobs get-run <RUN_ID> (poll .state.life_cycle_state)
Fetch run outputdatabricks jobs get-run-output <TASK_RUN_ID>
List clustersdatabricks clusters list --output json
Get cluster detailsdatabricks clusters get <CLUSTER_ID>
Start / restart / terminate clusterdatabricks clusters start/restart/delete <CLUSTER_ID>
Permanently delete clusterdatabricks clusters permanent-delete <CLUSTER_ID>
Create clusterdatabricks clusters create --json '{...}' (see 3-interactive-cluster.md)
List node types / Spark versionsdatabricks clusters list-node-types / databricks clusters spark-versions
Execute code on a running clusterdatabricks api post /api/1.2/contexts/create + databricks api post /api/1.2/commands/execute (see 3-interactive-cluster.md)
SQL warehousesdatabricks warehouses create/list/get/start/stop/edit/delete (see SQL Warehouses below)

SQL Warehouses

All ID-taking commands use positional arg (no --id flag). Use databricks warehouses list to find an ID.

# Create a serverless SQL warehouse. min_num_clusters + max_num_clusters are REQUIRED
# (the server rejects the default 0). Keep the aidevkit_project tag for resource tracking.
databricks warehouses create --json '{
  "name": "my-warehouse",
  "cluster_size": "Small",
  "enable_serverless_compute": true,
  "auto_stop_mins": 10,
  "min_num_clusters": 1,
  "max_num_clusters": 1,
  "tags": {"custom_tags": [{"key": "aidevkit_project", "value": "ai-dev-kit"}]}
}'

# List / find — trim to id, name, state with jq
databricks warehouses list -o json | jq '.[] | {id, name, state, size: .cluster_size}'

# Find by name
databricks warehouses list -o json | jq '.[] | select(.name == "my-warehouse")'

# Get one warehouse's full config
databricks warehouses get <WAREHOUSE_ID>

# Start / stop (both are LROs; add --no-wait to return immediately)
databricks warehouses start <WAREHOUSE_ID>
databricks warehouses stop  <WAREHOUSE_ID>

# Resize / reconfigure — pass the FULL desired config (omitted fields revert to defaults,
# so always re-state min_num_clusters/max_num_clusters). Use --no-wait if the warehouse
# is STOPPED, otherwise edit blocks trying to reach RUNNING and errors out (the mutation
# itself still applies). When the warehouse is already RUNNING, --no-wait is optional.
databricks warehouses edit <WAREHOUSE_ID> --no-wait --json '{
  "name": "my-warehouse",
  "cluster_size": "Medium",
  "enable_serverless_compute": true,
  "auto_stop_mins": 15,
  "min_num_clusters": 1,
  "max_num_clusters": 1
}'

# Delete (irreversible)
databricks warehouses delete <WAREHOUSE_ID>

Sizes: 2X-Small, X-Small, Small, Medium, Large, X-Large, 2X-Large, 3X-Large, 4X-Large. Types: set "warehouse_type": "PRO" (default) or "CLASSIC" in the JSON body.

Related Skills

Databricks Genie Agents

A Genie Agent is a curated agent scoped to one business area: its tables / Metric Views, documents (UC volumes), sample questions, and instructions. It answers by writing SQL on a warehouse and, in Agent mode, by searching its documents, so one turn can combine SQL results and cited files. Distinct from Genie One (databricks-data-discovery), which answers across your data without a curated Agent.

Replaces Knowledge Assistant and Supervisor Agent for docs + data. Both are legacy ("For new low-code agents over your documents and data, use Genie Agents"). Document Q&A (was a KA) = an Agent with a volume: no endpoint, no sync. Routing between data and docs (was a Supervisor over Genie + KA) = one Agent with tables + a volume. See Documents. An Agent can also attach Databricks-provided MCP connectors (Google Drive, Microsoft 365, Slack, Jira/Confluence, GitHub, ...; Beta, set up in the UI, Agent mode only). Custom MCP servers, UC functions, or other endpoints aren't supported; build a code agent (databricks-ml-training). To maintain an existing KA / Supervisor: legacy-ka-supervisor.md.

Genie Agent Lifecycle

PhaseReferenceLoad whenTypical CLI
Create / updatecreate-genie-agent.mdBuilding or editing an Agent: the workflow, the serialized_space payload format, and the CLIdiscover-schema → create-space / update-space
Attach documentscreate-genie-agent.md § DocumentsDocument Q&A, docs + data, replacing a KA / Supervisorvolumes create → update-space → probe in Agent mode
Query / converse / validatequery-genie-agent.mdConversation API (tables), Agent mode API (needed for documents), Metric View SQLstart-conversation / Agent mode responses
Improve (inspect / diagnose / benchmark / fix)improve-genie-agent.mdWorking with an existing Agent: wrong/empty answers, quality tuning, benchmark evaluationget-space; genie-create-eval-run; update-space
Export / migrategenie-agent-cicd.mdExport, import, cross-workspace migration, batch migration, DABs/CI-CDget-space → remap → create-space
Optimization historyuc-persistence.mdSetting up UC Delta tables for multi-pass optimization history (CREATE TABLE DDL only)
Legacylegacy-ka-supervisor.mdMaintaining or migrating an existing Knowledge Assistant / Supervisor Agentknowledge-assistants / supervisor-agents

Typical flow: create → query/validate → improve (diagnose / benchmark / fix, as needed).

Commands

An Agent is defined by a serialized_space JSON object (sources, sample questions, per-column context, instructions), kept in a local genie_agent.json and passed to the CLI. Core commands below; the lifecycle table points to the full workflow, payload schema, and flags.

databricks genie list-spaces                                          # find Agents
databricks genie get-space SPACE_ID --include-serialized-space        # inspect config
databricks genie create-space WAREHOUSE_ID "$(jq -c . genie_agent.json)" \
  --title "..." --parent-path /Workspace/Users/you@company.com/genie_spaces   # create
databricks genie update-space SPACE_ID --serialized-space "$(jq -c . genie_agent.json)"  # full-replace edit
databricks genie start-conversation --no-wait SPACE_ID "a question"   # converse / validate (tables only)
databricks genie trash-space SPACE_ID                                 # delete

Documents (replaces KA and Supervisor)

Attach a managed UC volume in data_sources.volumes[]: create-space / update-space start indexing automatically, and documents are read in Agent mode only (the chat-mode start-conversation ignores them). Setup, waiting for indexing, adding documents to an existing Agent, permissions, limits, and evaluation: create-genie-agent.md § Documents.

Prerequisites

  • Tables / Metric Views in Unity Catalog and/or a managed UC volume of documents
  • SQL Warehouse to execute queries (auto-detected if not specified)

Related Skills

Apache Iceberg on Databricks

Databricks provides multiple ways to work with Apache Iceberg: native managed Iceberg tables, UniForm for Delta-to-Iceberg interoperability, and the Iceberg REST Catalog (IRC) for external engine access.


Critical Rules (always follow)

  • MUST use Unity Catalog — all Iceberg features require UC-enabled workspaces
  • MUST NOT install an Iceberg library into Databricks Runtime (DBR includes built-in Iceberg support; adding a library causes version conflicts)
  • MUST NOT set write.metadata.path or write.metadata.previous-versions-max — Databricks manages metadata locations automatically; overriding causes corruption
  • MUST determine which Iceberg pattern fits the use case before writing code — see the When to Use section below
  • MUST know that both PARTITIONED BY and CLUSTER BY produce the same Iceberg metadata for external engines — UC maintains an Iceberg partition spec with partition fields corresponding to the clustering keys, so external engines reading via IRC see a partitioned Iceberg table (not Hive-style, but proper Iceberg partition fields) and can prune on those fields; internally UC uses those fields as liquid clustering keys; the only differences between the two syntaxes are: (1) PARTITIONED BY is standard Iceberg DDL (any engine can create the table), while CLUSTER BY is DBR-only DDL; (2) PARTITIONED BY auto-handles DV/row-tracking properties, while CLUSTER BY requires manual TBLPROPERTIES on v2
  • MUST NOT use expression-based partition transforms (bucket(), years(), months(), days(), hours()) with PARTITIONED BY on managed Iceberg tables — only plain column references are supported; expression transforms cause errors
  • MUST disable deletion vectors and row tracking when using CLUSTER BY on Iceberg v2 tables — set 'delta.enableDeletionVectors' = false and 'delta.enableRowTracking' = false in TBLPROPERTIES (Iceberg v3 handles this automatically; PARTITIONED BY handles this automatically on both v2 and v3)

Key Concepts

ConceptSummary
Managed Iceberg TableNative Iceberg table created with USING ICEBERG — full read/write in Databricks and via external Iceberg engines
External Iceberg Reads (Uniform)Delta table that auto-generates Iceberg metadata — read as Iceberg externally, write as Delta internally
Compatibility ModeUniForm variant for streaming tables and materialized views in SDP pipelines
Iceberg REST Catalog (IRC)Unity Catalog's built-in REST endpoint implementing the Iceberg REST Catalog spec — lets external engines (Spark, PyIceberg, Snowflake) access UC-managed Iceberg data
Iceberg v3Next-gen format (DBR 18 LTS+) — deletion vectors, VARIANT type, row lineage

Quick Start

Create a Managed Iceberg Table

-- No clustering
CREATE TABLE my_catalog.my_schema.events
USING ICEBERG
AS SELECT * FROM raw_events;

-- PARTITIONED BY (recommended for cross-platform): standard Iceberg syntax, works on EMR/OSS Spark/Trino/Flink
-- auto-disables DVs and row tracking — no TBLPROPERTIES needed on v2 or v3
CREATE TABLE my_catalog.my_schema.events
USING ICEBERG
PARTITIONED BY (event_date)
AS SELECT * FROM raw_events;

-- CLUSTER BY on Iceberg v2 (DBR-only syntax): must manually disable DVs and row tracking
CREATE TABLE my_catalog.my_schema.events
USING ICEBERG
TBLPROPERTIES (
  'delta.enableDeletionVectors' = false,
  'delta.enableRowTracking' = false
)
CLUSTER BY (event_date)
AS SELECT * FROM raw_events;

-- CLUSTER BY on Iceberg v3 (DBR-only syntax): no TBLPROPERTIES needed
CREATE TABLE my_catalog.my_schema.events
USING ICEBERG
TBLPROPERTIES ('format-version' = '3')
CLUSTER BY (event_date)
AS SELECT * FROM raw_events;

Enable UniForm on an Existing Delta Table

ALTER TABLE my_catalog.my_schema.customers
SET TBLPROPERTIES (
  'delta.columnMapping.mode' = 'name',
  'delta.enableIcebergCompatV2' = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg'
);

Read/Write Capability Matrix

Table TypeDatabricks ReadDatabricks WriteExternal IRC ReadExternal IRC Write
Managed Iceberg (USING ICEBERG)YesYesYesYes
Delta + UniFormYes (as Delta)Yes (as Delta)Yes (as Iceberg)No
Delta + Compatibility ModeYes (as Delta)YesYes (as Iceberg)No

Reference Files

FileSummaryKeywords
references/1-managed-iceberg-tables.mdCreating and managing native Iceberg tables — DDL, DML, Liquid Clustering, Predictive Optimization, Iceberg v3, limitationsCREATE TABLE USING ICEBERG, CTAS, MERGE, time travel, deletion vectors, VARIANT
references/2-uniform-and-compatibility.mdMaking Delta tables readable as Iceberg — UniForm for regular tables, Compatibility Mode for streaming tables and MVsUniForm, universalFormat, Compatibility Mode, streaming tables, materialized views, SDP
references/3-iceberg-rest-catalog.mdExposing Databricks tables to external engines via the IRC endpoint — auth, credential vending, IP access listsIRC, REST Catalog, credential vending, EXTERNAL USE SCHEMA, PAT, OAuth
references/4-snowflake-interop.mdBidirectional Snowflake-Databricks integration — catalog integration, foreign catalogs, vended credentialsSnowflake, catalog integration, external volume, vended credentials, REFRESH_INTERVAL_SECONDS
references/5-external-engine-interop.mdConnecting PyIceberg, OSS Spark, AWS EMR, Apache Flink, and Kafka Connect via IRCPyIceberg, OSS Spark, EMR, Flink, Kafka Connect, pyiceberg.yaml

When to Use


Common Issues

IssueSolution
No Change Data Feed (CDF)CDF is not supported on managed Iceberg tables. Use Delta + UniForm if you need CDF.
UniForm async delayIceberg metadata generation is asynchronous. After a write, there may be a brief delay before external engines see the latest data. Check status with DESCRIBE EXTENDED table_name.
Compression codec changeManaged Iceberg tables use zstd compression by default (not snappy). Older Iceberg readers that don't support zstd will fail. Verify reader compatibility or set write.parquet.compression-codec to snappy.
Snowflake 1000-commit limitSnowflake's Iceberg catalog integration can only see the last 1000 Iceberg commits. High-frequency writers must compact metadata or Snowflake will lose visibility of older data.
Deletion vectors with UniFormUniForm requires deletion vectors to be disabled (delta.enableDeletionVectors = false). If your table has deletion vectors enabled, disable them before enabling UniForm.
No shallow clone for IcebergSHALLOW CLONE is not supported for Iceberg tables. Use DEEP CLONE or CREATE TABLE ... AS SELECT instead.
Version mismatch with external enginesEnsure external engines use an Iceberg library version compatible with the format version of your tables. Iceberg v3 tables require Iceberg library 1.9.0+.

Related Skills

  • databricks-unity-catalog — catalog/schema management, governance, system tables
  • databricks-pipelines — SDP pipelines (streaming tables, materialized views with Compatibility Mode)
  • databricks-python-sdk — Python SDK and REST API for Databricks operations
  • databricks-dbsql — SQL warehouse features, query patterns

Resources

Lakeflow Jobs Development

FIRST: Use the parent databricks-core skill for CLI basics, authentication, profile selection, and data exploration commands.

Compute: jobs run on serverless by default. Before you choose compute or write any cluster setting, use the databricks-serverless skill; it decides serverless or classic and the performance mode.

Lakeflow Jobs orchestrate data workflows with multi-task DAGs, flexible triggers, and comprehensive monitoring. Jobs support diverse task types and can be managed via Declarative Automation Bundles (DABs), Python SDK, or CLI.

Reference Files

Use CaseReference File
Configure task types (notebook, Python, SQL, dbt, pipeline, JAR, run_job, for_each)references/task-types.md
Set up triggers and schedules (cron, periodic, file arrival, table update, continuous)references/triggers-schedules.md
Configure notifications, health rules, retries, timeouts, queuesreferences/notifications-monitoring.md
Complete worked examples (ETL, warehouse refresh, event-driven, ML training, multi-env, streaming, cross-job)references/examples.md

Scaffolding a New Job Project

Use databricks bundle init with a config file to scaffold non-interactively. This creates a project in the <project_name>/ directory:

databricks bundle init default-python --config-file <(echo '{"project_name": "my_job", "include_job": "yes", "include_pipeline": "no", "include_python": "yes", "serverless": "yes"}') --profile <PROFILE> < /dev/null
  • project_name: letters, numbers, underscores only

bundle init also creates an AGENTS.md in the project. Customize that file for the project instead of creating new agent instruction files.

Project Structure

my-job-project/
├── databricks.yml              # Bundle configuration
├── resources/
│   └── my_job.job.yml          # Job definition
├── src/
│   ├── my_notebook.ipynb       # Notebook tasks
│   └── my_module/              # Python wheel package
│       ├── __init__.py
│       └── main.py
├── tests/
│   └── test_main.py
└── pyproject.toml              # Python project config (if using wheels)

Quick Start

Declarative Automation Bundles (DABs) — recommended

# resources/jobs.yml
resources:
  jobs:
    my_etl_job:
      name: "[${bundle.target}] My ETL Job"
      tasks:
        - task_key: extract
          notebook_task:
            notebook_path: ../src/notebooks/extract.py
      # no cluster block = serverless; compute details: databricks-serverless

Python SDK

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.jobs import Task, NotebookTask, Source

w = WorkspaceClient()

job = w.jobs.create(
    name="my-etl-job",
    tasks=[
        Task(
            task_key="extract",
            notebook_task=NotebookTask(
                notebook_path="/Workspace/Shared/etl/extract",
                source=Source.WORKSPACE,
            ),
        ),
    ],
)
print(f"Created job: {job.job_id}")

CLI

databricks jobs create --json '{
  "name": "my-etl-job",
  "tasks": [{
    "task_key": "extract",
    "notebook_task": {
      "notebook_path": "/Workspace/Shared/etl/extract",
      "source": "WORKSPACE"
    }
  }]
}'

Core Concepts

Multi-Task Workflows

Jobs support DAG-based task dependencies:

tasks:
  - task_key: extract
    notebook_task:
      notebook_path: ../src/extract.py

  - task_key: transform
    depends_on:
      - task_key: extract
    notebook_task:
      notebook_path: ../src/transform.py

  - task_key: load
    depends_on:
      - task_key: transform
    run_if: ALL_SUCCESS  # Only run if all dependencies succeed
    notebook_task:
      notebook_path: ../src/load.py

run_if conditions:

  • ALL_SUCCESS (default) — run when all dependencies succeed
  • ALL_DONE — run when all dependencies complete (success or failure)
  • AT_LEAST_ONE_SUCCESS — run when at least one dependency succeeds
  • NONE_FAILED — run when no dependencies failed
  • ALL_FAILED — run when all dependencies failed
  • AT_LEAST_ONE_FAILED — run when at least one dependency failed

Task Types Summary

Task TypeUse CaseReference
notebook_taskRun notebooksreferences/task-types.md#notebook-task
spark_python_taskRun Python scriptsreferences/task-types.md#spark-python-task
python_wheel_taskRun Python wheelsreferences/task-types.md#python-wheel-task
sql_taskRun SQL queries/files/dashboards/alertsreferences/task-types.md#sql-task
dbt_taskRun dbt projectsreferences/task-types.md#dbt-task
pipeline_taskTrigger SDP (formerly DLT) pipelinesreferences/task-types.md#pipeline-task
spark_jar_taskRun Spark JARsreferences/task-types.md#spark-jar-task
run_job_taskTrigger other jobsreferences/task-types.md#run-job-task
for_each_taskLoop over inputsreferences/task-types.md#for-each-task

Trigger Types Summary

Trigger TypeUse CaseReference
scheduleCron-based schedulingreferences/triggers-schedules.md#cron-schedule
trigger.periodicInterval-basedreferences/triggers-schedules.md#periodic-trigger
trigger.file_arrivalFile arrival eventsreferences/triggers-schedules.md#file-arrival-trigger
trigger.table_updateUnity Catalog table change eventsreferences/triggers-schedules.md#table-update-trigger
continuousAlways-running jobsreferences/triggers-schedules.md#continuous-jobs

Compute Configuration

Serverless is the default. Leave out cluster settings (job_clusters, new_cluster, job_cluster_key, existing_cluster_id); a notebook, Python script, wheel or JAR task without them runs on serverless. SQL tasks, and the SQL of dbt tasks, run on a SQL warehouse (warehouse_id, see Task Types); a pipeline task runs on the pipeline's own compute (serverless: true in the pipeline), and the job's performance_target sets its mode. If creating or running the job fails because serverless is not available in the workspace, put the whole job on classic and say so (job clusters for the tasks that run on job compute).

For everything else about compute, use the databricks-serverless skill: the performance mode (performance_target), libraries (environments), existing code that may not run on serverless, and the few workloads that need a classic cluster. If that skill is not installed, run databricks aitools install. Either way, set performance_target on every job: STANDARD to optimize cost, PERFORMANCE_OPTIMIZED when the job is time-sensitive, interactive, or other workloads depend on it.

Job Parameters

Parameters defined at job level are passed to ALL tasks (no need to repeat per task):

parameters:
  - name: env
    default: "dev"
  - name: date
    default: "{{start_date}}"  # Dynamic value reference

Access in notebooks:

catalog = dbutils.widgets.get("env")
load_date = dbutils.widgets.get("date")

Pass to specific tasks:

tasks:
  - task_key: my_task
    notebook_task:
      notebook_path: ../src/notebook.py
      base_parameters:
        env: "{{job.parameters.env}}"
        custom_param: "value"

Common Operations

Python SDK

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

# List jobs
jobs = w.jobs.list()

# Get job details
job = w.jobs.get(job_id=12345)

# Run job now
run = w.jobs.run_now(job_id=12345)

# Run with parameters
run = w.jobs.run_now(
    job_id=12345,
    job_parameters={"env": "prod", "date": "2024-01-15"},
)

# Cancel run
w.jobs.cancel_run(run_id=run.run_id)

# Delete job
w.jobs.delete(job_id=12345)

CLI

# List jobs
databricks jobs list

# Get job details
databricks jobs get 12345

# Run job
databricks jobs run-now 12345

# Run with parameters (must use --json with job_id inside)
databricks jobs run-now --json '{"job_id": 12345, "job_parameters": {"env": "prod"}}'

# Cancel run
databricks jobs cancel-run 67890

# Delete job
databricks jobs delete 12345

Declarative Automation Bundle Operations

# Validate configuration
databricks bundle validate --profile <profile>

# Deploy to a target
databricks bundle deploy -t dev --profile <profile>

# Run a job
databricks bundle run <job_name> -t dev --profile <profile>

# Check run status
databricks jobs get-run --run-id <id> --profile <profile>

# Destroy resources
databricks bundle destroy --auto-approve

Permissions (DABs)

resources:
  jobs:
    my_job:
      name: "My Job"
      permissions:
        - level: CAN_VIEW
          group_name: "data-analysts"
        - level: CAN_MANAGE_RUN
          group_name: "data-engineers"
        - level: CAN_MANAGE
          user_name: "admin@example.com"

Permission levels:

  • CAN_VIEW — view job and run history
  • CAN_MANAGE_RUN — view, trigger, and cancel runs
  • CAN_MANAGE — full control including edit and delete

Unit Testing

Run unit tests locally:

uv run pytest

Development Workflow

  1. Validate: databricks bundle validate --profile <profile>
  2. Deploy: databricks bundle deploy -t dev --profile <profile>
  3. Run: databricks bundle run <job_name> -t dev --profile <profile>
  4. Check run status: databricks jobs get-run --run-id <id> --profile <profile>

Common Issues

IssueSolution
Serverless job startup slowperformance_target: PERFORMANCE_OPTIMIZED; see databricks-serverless
Task dependencies not workingVerify task_key references match exactly in depends_on
Schedule not triggeringCheck pause_status: UNPAUSED and valid timezone
File arrival not detectingEnsure path has proper permissions and uses cloud storage URL
Table update trigger missing eventsVerify Unity Catalog table and proper grants
Parameter not accessibleUse dbutils.widgets.get() in notebooks
admins group errorCannot modify admins permissions on jobs
Serverless task failsSee the Common Issues of databricks-serverless (libraries, RDD code, DBFS mounts)

Related Skills

  • databricks-dabs — DABs configuration patterns shared by jobs and pipelines
  • databricks-pipelines — SDP (formerly DLT) pipelines triggered by pipeline_task
  • databricks-serverless — compute decisions: serverless, performance mode, classic fallback

Documentation

Lakebase Postgres Autoscaling

FIRST: Use the parent databricks-core skill for CLI basics, authentication, and profile selection.

Lakebase is Databricks' serverless Postgres-compatible database, available on AWS and Azure (GA) and on GCP (Beta). It provides fully managed OLTP storage with autoscaling, branching, and scale-to-zero.

Regions: Lakebase runs only in the commercial regions listed in Region availability (check the page for the user's cloud). It isn't available on AWS GovCloud, and the Azure list has no Azure Government or Azure China regions. If the workspace is in one of these clouds or an unlisted region, tell the user that Lakebase isn't available there. Don't try to create a project.

Provisioned Lakebase no longer exists. Never create it. The old Provisioned tier (static, non-scalable Postgres; databricks database CLI / w.database SDK) is being migrated to Autoscaling over summer 2026, with no customer action required. Lakebase today is Autoscaling-only (databricks postgres / w.postgres). Do not attempt to create Provisioned instances; if a user asks for one, tell them Provisioned instances don't exist anymore and point them to Autoscaling. Treat the legacy database resource key in app configs as retired, and migrate to the postgres resource key. For the legacy DAB synced_database_tables resource, use the postgres_synced_tables bundle resource or the databricks postgres create-synced-table CLI flow instead (see the Common Issues row and references/synced-tables.md).

For context when reading older configs or docs, here is how the retired Provisioned tier maps to Autoscaling:

AspectProvisioned (retired)Autoscaling (current)
CLI groupdatabricks databasedatabricks postgres
Top-level resourceInstanceProject
CapacityCU_1–CU_8 (16 GB/CU)0.5–112 CU (2 GB/CU)
BranchingNot supportedFull support
Scale-to-zeroNot supportedConfigurable
HAReadable secondaries1–3 secondaries + read replicas
Data APINot availablePostgREST HTTP API
CloudAWS onlyAWS and Azure (GA), GCP (Beta)

Compliance: Supports HIPAA, C5, TISAX, or None. Before any PHI goes into Lakebase, apply the HIPAA gate below.

HIPAA Gate (PHI)

HIPAA coverage comes from the workspace: the compliance security profile with the HIPAA standard, plus a Business Associate Agreement. Confirm both with the user before PHI goes in. Then:

  • Preview features: do not use a Preview or Beta feature (for example, Lakehouse Sync) for PHI unless it is listed for your region in Regional support for features. Check the list each time.
  • Existing projects: a project that was running when HIPAA was turned on has no HIPAA audit logging until its compute restarts. Databricks does not restart it for you. Restart each compute.
  • No PHI in metadata: never put PHI in database, schema, table, column, or role names, in tags, or in other customer-defined fields. These appear in audit logs and system metadata, and Databricks cannot remove them later.
  • No PHI in SQL text: pass PHI values as query parameters, never as literals. Audit logs record statement text.
  • No PHI in logs or tickets: keep PHI out of app logs, error messages, screenshots, diagnostic output, and support requests.
  • Least privilege: grant each role only the schemas, tables, and operations it needs.

Sources: HIPAA compliance for Lakebase, Enable HIPAA compliance.

Row-Level Security and Data API: Must-Follow Rules

Apply these whenever a design has per-user rows, RLS, or the Data API. Details and tests are in connectivity.md.

  • Many end users: a branch holds at most 500 Postgres roles. Never create one role per app user. The app connects as its service principal over a pool and sets the user per transaction: SELECT set_config('app.user', $1, true).
  • RLS on a setting: wrap it in NULLIF: USING (owner_email = (SELECT NULLIF(current_setting('app.user', true), ''))). On a reused pooled connection, an unset value reads '', not NULL. Also run ALTER TABLE ... FORCE ROW LEVEL SECURITY, because the table owner bypasses RLS otherwise.
  • Testing RLS: never test as the project owner or databricks_superuser, and never grant databricks_superuser to a teammate for local work. Both bypass RLS. Use a role with only USAGE on the app's schema and the table privileges it needs.
  • Views over RLS tables: create them WITH (security_invoker = true), and grant the caller SELECT on the base table too. A default view checks RLS as its owner, so every caller sees all rows.
  • Data API before production: the defaults are open. Empty CORS origins means all origins, and empty maximum rows means no limit. Set the exact app origins and a row limit, and expose only the schema built for the API, not public.

Capabilities

  • Project lifecycle -- create, update, delete Lakebase Postgres Autoscaling projects
  • Branching -- copy-on-write branches with TTL, point-in-time recovery, and reset
  • Compute scaling -- autoscale 0.5--64 CU, fixed up to 112 CU, scale-to-zero
  • High availability -- 1 primary + 1--3 secondaries, automatic failover
  • PostgreSQL connectivity -- OAuth token refresh, connection pooling, SSL
  • Data API -- PostgREST-compatible HTTP CRUD (Autoscaling only)
  • Lakebase synced tables -- sync Unity Catalog Delta tables into Postgres (previously known as Reverse ETL)
  • Databricks App integration -- scaffold apps with Lakebase feature, deploy-first workflow
  • Cloud support -- AWS and Azure (GA), GCP (Beta)

Reference docs:

  • computes-and-scaling.md — Sizing, endpoint management, scale-to-zero, HA
  • connectivity.md — Connection patterns, token refresh, Data API
  • synced-tables.md — Lakebase synced tables, data type mapping, capacity planning
  • lakehouse-sync.md — CDC from Lakebase Postgres to Unity Catalog Delta tables (programmable via databricks postgres *-cdf-config / w.postgres.*_cdf_config, Beta)
  • pgvector.md — Vector similarity search with pgvector extension
  • off-platform.md — Off-platform Lakebase (NOT Databricks Apps): external Node.js apps connecting via @databricks/lakebase, env management, token refresh, Drizzle ORM

Resource Hierarchy

Project (top-level container)
  └── Branch (isolated database environment, copy-on-write)
        ├── Endpoint (read-write or read-only)
        ├── Database (standard Postgres DB)
        └── Role (Postgres role)
  • Project: Top-level container. Creating one auto-provisions a production branch and a primary read-write endpoint.
  • Branch: Isolated database environment sharing storage with parent (copy-on-write). States: READY, ARCHIVED.
  • Endpoint (called Compute in UI): Compute resource powering a branch. Types: ENDPOINT_TYPE_READ_WRITE, ENDPOINT_TYPE_READ_ONLY.
  • Database: Standard Postgres database within a branch. Default: databricks_postgres.
  • Role: Postgres role within a branch.

Resource Name Formats

ResourceFormat
Projectprojects/{project_id}
Branchprojects/{project_id}/branches/{branch_id}
Endpointprojects/{project_id}/branches/{branch_id}/endpoints/{endpoint_id}
Databaseprojects/{project_id}/branches/{branch_id}/databases/{database_id}

All IDs: 1-63 characters, start with lowercase letter, lowercase letters/numbers/hyphens only (RFC 1123).

CLI Discovery -- ALWAYS Do This First

Note: "Lakebase" is the product name; the CLI command group is postgres. All commands use databricks postgres ....

Do NOT guess command syntax. Discover available commands dynamically:

databricks postgres -h                    # List all subcommands
databricks postgres <subcommand> -h       # Flags, args, JSON fields

Create a Project

First decide: reuse or create. When building or attaching to an app, ask the user whether to reuse an existing project/branch/database — list them with databricks postgres list-projects (then list-branches / list-databases), let the user pick, and confirm which schema the app will own — or create a new project. Only skip listing and create directly when the user explicitly asked for a brand-new project.

databricks postgres create-project <PROJECT_ID> \
  --json '{"spec": {"display_name": "<DISPLAY_NAME>"}}' \
  --profile <PROFILE>

Auto-creates: production branch + primary read-write endpoint with scale-to-zero on (24 h timeout). The documented default size is 8–16 CU (autoscaling), but a project created with the CLI can get a 1 CU min/max endpoint instead. Check the size with get-endpoint and resize with update-endpoint if the workload needs more. Long-running operation; CLI waits by default. Use --no-wait to return immediately.

After creation, verify:

databricks postgres list-branches projects/<PROJECT_ID> --profile <PROFILE>
databricks postgres list-endpoints projects/<PROJECT_ID>/branches/<BRANCH_ID> --profile <PROFILE>
databricks postgres list-databases projects/<PROJECT_ID>/branches/<BRANCH_ID> --profile <PROFILE>

Extract connection values from JSON output:

ValueJSON pathUsed for
Endpoint hoststatus.hosts.hostPGHOST, lakebase.postgres.host
Endpoint resource pathnameLAKEBASE_ENDPOINT, lakebase.postgres.endpointPath
Database resource pathnamelakebase.postgres.database
PostgreSQL database namestatus.postgres_databasePGDATABASE, lakebase.postgres.databaseName

Updating a Project

databricks postgres update-project projects/<PROJECT_ID> spec.display_name \
  --json '{"spec": {"display_name": "My Updated Application"}}' \
  --profile <PROFILE>

Deleting a Project

WARNING: Permanent -- deletes all branches, computes, databases, roles, and data. Do not delete without explicit user permission.

databricks postgres delete-project projects/<PROJECT_ID> --profile <PROFILE>

Autoscaling

Endpoints use compute units (CU) (~2 GB RAM per CU). Range: 0.5--64 CU (autoscaling or fixed), 65--112 CU (fixed only). Scale-to-zero enabled by default on every branch, including production (24 h timeout; computes ≤ 32 CU only).

See computes-and-scaling.md for sizing tables, endpoint CRUD, and configuration details.

Branches

Branches are copy-on-write snapshots. Use for testing schema migrations, trying queries, or previewing data changes without affecting production.

databricks postgres create-branch projects/<PROJECT_ID> <BRANCH_ID> \
  --json '{"spec": {"source_branch": "projects/<PROJECT_ID>/branches/<SOURCE>", "no_expiry": true}}' \
  --profile <PROFILE>

Branches require an expiration policy: "no_expiry": true for permanent, or "ttl": "<seconds>s" (max 30 days).

Limits: 500 branches per project. Per-branch storage quota is adjustable on request (contact Databricks Support). 500 Postgres roles per branch. 1,000 projects per workspace. For apps with many end users, do not create one role per user; see "Many Users: Roles and RLS at Scale" in connectivity.md.

Use CaseTTL
CI/CD environments2--4 hours ("ttl": "14400s")
Demos24--48 hours ("ttl": "172800s")
Feature development1--7 days ("ttl": "604800s")
Long-term testingUp to 30 days ("ttl": "2592000s")

Point-in-time branching: Create from a past state (within restore window) for recovery. The time specification fields live inside the --json spec (spec.source_branch_time, spec.source_branch_lsn), not as top-level create-branch flags.

Reset: Replaces branch data and schema with the latest from its parent. Local changes are lost. Root branches and branches with children cannot be reset; protected branches (spec.is_protected) cannot be reset either.

Reset is UI-only — there is no databricks postgres reset-branch CLI command and no API/SDK method for it. In the Lakebase App, go to the project's Branches page, click the menu next to the branch, and select Reset from parent.

CLI workarounds:

  • Delete + recreate: delete-branch, then create-branch with spec.source_branch set to the parent (unprotect via update-branch first if protected).
  • Point-in-time: create-branch with spec.source_branch_time or spec.source_branch_lsn. Timestamp-based reset is not supported; use PITR branching instead.

Delete: Protected branches must be unprotected first (update-branch to set spec.is_protected to false). Cannot delete branches with children. Never delete the production branch.

What's Next

Build a Databricks App

After creating a project, scaffold a connected Databricks App:

# 1. Get branch name
databricks postgres list-branches projects/<PROJECT_ID> --profile <PROFILE>

# 2. Get database name
databricks postgres list-databases projects/<PROJECT_ID>/branches/<BRANCH_ID> --profile <PROFILE>

# 3. Scaffold with lakebase feature
databricks apps init --name <APP_NAME> --features lakebase \
  --set "lakebase.postgres.branch=<BRANCH_NAME>" \
  --set "lakebase.postgres.database=<DATABASE_NAME>" \
  --run none --profile <PROFILE>

For the full app workflow, use the databricks-apps skill.

Attach Lakebase to an existing app

apps init --features lakebase (above) wires the database at scaffold time. To attach a project to an existing app, update its resources.

Use the postgres resource key — its fields are branch + database (full resource paths from the table above). The legacy database key (instance_name + database_name) is deprecated; using it fails with Database instance <name> does not exist. Get the exact paths from list-branches / list-databases (the DB name is often hyphenated, e.g. databricks-postgres).

Update the app's resources with databricks apps create-update — the method to use for any app (the older databricks apps update is legacy and can't change resources for an app in a space). update_mask=resources replaces the whole resources array, so read the app's current resources and merge the new one in (or you'll detach the rest). Pass everything in --json; only APP_NAME is positional:

databricks apps create-update <APP_NAME> --json @update.json --profile <PROFILE>   # waits for completion; --no-wait to return early
{
  "update_mask": "resources",
  "app": {
    "resources": [
      {
        "name": "postgres",
        "postgres": {
          "branch": "projects/<PROJECT_ID>/branches/<BRANCH_ID>",
          "database": "projects/<PROJECT_ID>/branches/<BRANCH_ID>/databases/<DATABASE_ID>",
          "permission": "CAN_CONNECT_AND_CREATE"
        }
      }
    ]
  }
}

Confirm the branch/database with the user — don't default to production silently. The app's service principal must be able to create and own the schema(s) it uses there, so avoid a branch/database where those schema names are already owned by a user (the SP will hit permission denied … 42501) — a fresh/dedicated branch, or a new app-owned schema, is cleanest. See Schema Permissions for Deployed Apps below for the full ownership model.

Schema Permissions for Deployed Apps

The app's Service Principal has CAN_CONNECT_AND_CREATE -- it can create new objects but cannot access existing schemas. The SP must create the schema to become its owner.

ALWAYS deploy the app before running it locally. This is the #1 source of Lakebase permission errors.

Correct workflow:

  1. Deploy first: databricks apps deploy <APP_NAME> --profile <PROFILE>
  2. Grant local access (if needed): give your own identity a role with access to the app's schema only (SQL below). Do not assign databricks_superuser for this. Project creators already have access.
  3. Develop locally: your role can use only the schema and tables you granted
-- Run as the project owner. If a GRANT fails with "permission denied", run the GRANTs as the schema owner (the app's SP).
CREATE EXTENSION IF NOT EXISTS databricks_auth;
SELECT databricks_create_role('<YOUR_EMAIL>', 'USER');
GRANT USAGE ON SCHEMA <APP_SCHEMA> TO "<YOUR_EMAIL>";
GRANT SELECT, INSERT, UPDATE, DELETE ON <APP_SCHEMA>.<TABLE> TO "<YOUR_EMAIL>";   -- one line per table the app uses
GRANT USAGE ON SEQUENCE <SEQUENCE> TO "<YOUR_EMAIL>";   -- per SERIAL/identity column you insert into; name: SELECT pg_get_serial_sequence('<APP_SCHEMA>.<TABLE>', '<COLUMN>')
  • This role cannot create or change schemas or tables. The app's startup DDL fails with permission denied, even CREATE SCHEMA IF NOT EXISTS and CREATE TABLE IF NOT EXISTS on objects that already exist. The deployed app's SP creates and migrates the schema. Locally, skip the DDL when your role lacks CREATE on the schema (has_schema_privilege('<APP_SCHEMA>', 'CREATE')).
  • A table added later needs its own GRANT.

The project owner and databricks_superuser bypass RLS, even with FORCE ROW LEVEL SECURITY (both have BYPASSRLS). A query as either one returns every row, so it cannot show that a policy works. Test isolation as a role without BYPASSRLS: GRANT <ROLE> TO CURRENT_USER WITH SET TRUE;, then SET ROLE <ROLE>;, then run the app's queries.

If you already ran locally first and hit permission denied: the schema is owned by your credentials, not the SP. Do NOT drop the schema without asking the user -- dropping it deletes all data.

Ask the user to choose:

  • (A) Drop and redeploy: databricks psql --project <PROJECT_ID> -- -c "DROP SCHEMA IF EXISTS <SCHEMA_NAME> CASCADE;", then databricks apps deploy from the app directory. The SP recreates the schema on startup.
  • (B) Export first, then drop and redeploy: export via pg_dump (use connection details from databricks postgres get-endpoint; see Other Workflows below for HOST and TOKEN) or copy tables to a temp schema using databricks psql --project <PROJECT_ID>, then do option A. After the SP recreates the schema on redeploy, restore with pg_restore or re-INSERT from the temp schema.

Other Workflows

# Connect a Postgres client -- get connection string
databricks postgres get-endpoint projects/<PROJECT_ID>/branches/<BRANCH_ID>/endpoints/<ENDPOINT_ID> --profile <PROFILE>

# Manage roles
databricks postgres create-role -h

# Add a read replica
databricks postgres create-endpoint projects/<PROJECT_ID>/branches/<BRANCH_ID> <ENDPOINT_ID> \
  --json '{"spec": {"type": "ENDPOINT_TYPE_READ_ONLY"}}' --profile <PROFILE>

Run SQL against Lakebase (GRANT, CREATE INDEX, etc.):

# 1. Get endpoint host
databricks postgres get-endpoint projects/<PROJECT_ID>/branches/<BRANCH_ID>/endpoints/<ENDPOINT_ID> --profile <PROFILE>

# 2. Generate OAuth token
databricks postgres generate-database-credential \
  projects/<PROJECT_ID>/branches/<BRANCH_ID>/endpoints/<ENDPOINT_ID> \
  --profile <PROFILE>

# 3. Connect (use token from step 2 as password, host from step 1)
PGPASSWORD='<TOKEN>' psql "host=<HOST> user=<USERNAME> dbname=databricks_postgres sslmode=require"

Note: generate-database-credential requires the endpoint resource path (.../endpoints/<ENDPOINT_ID>), not a database or branch path.

Scriptable version (single copy-paste, useful for agents):

EP=projects/<PROJECT_ID>/branches/<BRANCH_ID>/endpoints/<ENDPOINT_ID>
# get-endpoint JSON shape: {"status": {"hosts": {"host": "<HOSTNAME>"}, ...}, ...}
HOST=$(databricks postgres get-endpoint $EP --profile <PROFILE> -o json \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['status']['hosts']['host'])")
TOKEN=$(databricks postgres generate-database-credential $EP --profile <PROFILE> -o json \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['token'])")
PGPASSWORD="$TOKEN" psql "host=$HOST user=<USERNAME> dbname=databricks_postgres sslmode=require"

Grant app SP access to synced tables (run after sync is ONLINE and the app is deployed, as the identity that created the synced table: it holds SELECT ... WITH GRANT OPTION, while databricks_writer_<n> owns the table):

-- <SCHEMA>.<SYNCED_TABLE>: the last two parts of the synced table ID <UC_CATALOG>.<SCHEMA>.<TABLE>
GRANT USAGE ON SCHEMA <SCHEMA> TO "<SP_CLIENT_ID>";
GRANT SELECT ON <SCHEMA>.<SYNCED_TABLE> TO "<SP_CLIENT_ID>";   -- one line per synced table; never ALL TABLES IN SCHEMA public (it also covers Lakebase internal objects)

Run the GRANT SELECT again after each new synced table reaches ONLINE. ALTER DEFAULT PRIVILEGES does not help here: it covers only tables that its own role creates later, and synced tables are owned by databricks_writer_<dbid>. For least-privilege, consider syncing into a dedicated schema instead of public so the grant is scoped to synced data only.

Get SP client ID: databricks apps get <APP_NAME> --profile <PROFILE> → service_principal_client_id field.

Data API: PostgREST-compatible HTTP CRUD on Postgres tables. See connectivity.md. Synced Tables: Sync Delta tables into Lakebase. See synced-tables.md.

PostgreSQL Extensions

Lakebase supports PostgreSQL extensions (e.g., pgvector for vector embeddings, pg_stat_statements for query statistics). See the full list of supported extensions.

-- List available extensions
SELECT * FROM pg_available_extensions ORDER BY name;

-- Install an extension
CREATE EXTENSION IF NOT EXISTS <extension_name>;

For vector embeddings with pgvector, see pgvector.md.

Troubleshooting

ErrorSolution
cannot configure default credentialsUse --profile flag or authenticate first
PERMISSION_DENIEDCheck workspace permissions
permission denied for schemaSchema owned by another role. If app not yet deployed: deploy first so the SP creates and owns the schema. If deployed but hitting this error (dev ran locally first): warn user about data loss, offer to export first (pg_dump with connection details from databricks postgres get-endpoint, or temp schema copy via databricks psql), then DROP SCHEMA IF EXISTS <SCHEMA_NAME> CASCADE + redeploy.
Protected branch won't deleteupdate-branch to set spec.is_protected to false first
Long-running operation timeoutUse --no-wait and poll with get-operation
New connections fail ~1 h after app startOAuth token was fetched once at startup. Expiry is checked only at login, so open connections keep working; refresh the token before it expires and pass the current one to each new connection (see token refresh)
Connection refused after scale-to-zeroCompute is waking up; retry with a short backoff (see scale-to-zero)
Branch deletion blockedDelete child branches first
Autoscaling range too wideMax - Min cannot exceed 16 CU
SSL required errorAlways use sslmode=require
Update mask requiredAll update-* operations require specifying fields (see -h)
Connection closed by serverConnections idle for 24 h are closed, and connections alive for more than 3 days may be closed regardless of activity. Use pool health checks and a max lifetime; implement retry.
DNS resolution fails (macOS)Python socket.getaddrinfo() fails with long hostnames. Use dig to resolve IP, pass via hostaddr param alongside host (for TLS SNI). See connectivity.md.
storage_catalog pipeline failurenew_pipeline_spec.storage_catalog must be a regular UC catalog, not the Lakebase catalog. DLT cannot write event logs to Postgres-backed schemas.
Synced table CDF errorTriggered/Continuous need a change data feed. Enable CDF on the source (ALTER TABLE ... SET TBLPROPERTIES (delta.enableChangeDataFeed = true)), use a source that Automatic change data feed supports, or use Snapshot mode.
Sync permissions errorEnsure USE CATALOG/USE SCHEMA on source table and CREATE TABLE in storage catalog
Synced table null bytesNull bytes (0x00) in STRING/ARRAY/MAP/STRUCT columns cause sync failures. Sanitize source data: REPLACE(col, CAST(CHAR(0) AS STRING), '')
Synced table data modifiedOnly read queries, indexes, and DROP TABLE allowed on synced tables in Postgres. Modifications break sync pipeline.
DABs synced_database_tablesDeprecated — maps to a legacy API and fails on current Lakebase. Use the postgres_synced_tables bundle resource (Beta, CLI v1.0.0+) or the databricks postgres create-synced-table CLI instead.

SDK and Version Requirements

ComponentMinimum Version
Databricks CLI>= v0.294.0
Databricks SDK for Python>= 0.81.0 (for w.postgres module)
psycopg2.x or 3.x (3.x recommended for async/pooling)
Postgres16, 17, or 18 (default: PG 17)

Lakeflow Connect: SaaS connectors

FIRST: use the parent databricks-core skill for CLI install, authentication, and profile selection.

SourceSkill
A SaaS application, a RabbitMQ queue, or a Kafka topic with the managed connector (Beta)this skill
A Kafka topic whose messages need more than the key and value formats the Kafka pipeline page listsread_kafka in databricks-pipelines, or databricks-spark-structured-streaming
A database (SQL Server, PostgreSQL, MySQL, Oracle, query-based)databricks-lakeflow-connect
An app or device that pushes eventsdatabricks-zerobus-ingest
Files on cloud object storageAuto Loader, databricks-pipelines
Files on an SFTP serverSFTP file connector on Auto Loader: file-connectors.md
A stream with no managed connector (Kinesis, Pub/Sub, Pulsar)databricks-spark-structured-streaming
A source with no managed connectora custom connector (Beta) or a community connector

Find the connector's pages

Connectors and options change faster than this skill: read the connector's docs pages instead of answering from memory. When a page and this skill disagree, the page wins.

  • Connector keys: the links on SaaS connectors, streaming connectors (Kafka, RabbitMQ) and file connectors. Say a connector isn't available only when none of them lists it.
  • Keys on those pages as of 2026-11 (newer ones may be there): 1password-event-logs, aha, akamai-waf, amplitude, anaplan, anthropic, anysphere-audit-logs, anysphere-organization, asana, atlassian-audit-logs, azure-devops, box-audit-logs, celigo, confluence, cribl, crowdstrike-falcon-event-stream, d365, github, gitlab, glean, gmail, google-ads, google-analytics, google-analytics-aggregated, google-drive, google-search-console, google-workspace, hubspot, jira, kafka, linkedin-ads, marketo, meta-ads, microsoft-365, microsoft-entra-id, microsoft-graph-logs, monday-com, netskope-logs, netsuite, notion, okta, okta-system-logs, onedrive, openai, oracle-eloqua, outlook, pagerduty, paypal, pendo, productboard, quickbooks, rabbitmq, reddit-ads, salesforce, sendgrid, servicenow, sfmc, sharepoint, shopify, smartsheet, square, strac, surveymonkey, tiktok-ads, veeva-vault, verkada, vimeo, wiz-audit-logs, workday-activity-logging, workday-hcm, workday-reports, workiva, xero, zendesk-support, zip, zoho-books, zoho-crm.
  • Pages: https://docs.databricks.com/ingestion/lakeflow-connect/<key> (or <key>-overview), then -source-setup, -connection, -pipeline, -reference, -limits, -troubleshoot (not every connector has each).
  • Read pages with the web-fetch tool or the databricks-docs skill. If neither is available, give the user the page links and ask what they show; don't fetch another way.
  • Read -reference before naming a source table or saying which tables are incremental; -limits before sizing.
  • Notes to read before the steps: Google, Meta, LinkedIn, Reddit, TikTok Ads → ads.md; SharePoint, Google Drive, OneDrive, SFTP → file-connectors.md; Kafka, RabbitMQ → streaming.md; the rest → connector-notes.md.

Workflow

Ask for every missing input in one message (connector, objects, destination catalog and schema, schedule), list what you'll create, and wait for one "yes".

  1. Previews toggle. When the page's Beta note mentions the Previews page, a workspace admin turns that preview on first. Beta has no production SLA: say so.
  2. Source setup. Follow -source-setup on the SaaS side (app registration, API user, scopes). Connectors with several auth methods have one setup page each.
  3. Connection. Created in Catalog Explorer (the -connection page has the exact path), often with a browser sign-in. Ask the user to create it from the -connection page and give you its name; don't guess connection_type or option names for databricks connections create. Grants: databricks-unity-catalog.
  4. Pipeline. Copy the object shape from the -pipeline page examples. source_schema is usually default, the account ID for ads, objects for Salesforce; never a placeholder.
  5. Run and schedule. Start one update and check it, then schedule a Jobs pipeline_task (databricks-pipelines, databricks-jobs). Anything kept: a bundle pipelines resource with this ingestion_definition plus a job with a pipeline_task (databricks-dabs). Kafka and RabbitMQ run continuously, with no job.
{
  "name": "jira_ingest",
  "catalog": "main",
  "schema": "jira_raw",
  "ingestion_definition": {
    "connection_name": "<jira-connection>",
    "objects": [
      {"table": {"source_schema": "default", "source_table": "issues",
                 "destination_catalog": "main", "destination_schema": "jira_raw"}}
    ]
  }
}
  • connection_name is the UC connection's name, not its ID.
  • objects[].table per table, objects[].schema for every table of a source schema, or objects[].report with source_url (Workday Reports).
  • connector_options carries connector settings (lookback windows, report levels, URLs); the keys don't follow the connector key (Google Drive is gdrive_options): take them from databricks bundle schema (pipelines.ConnectorOptions).
  • table_configuration carries per-table settings such as scd_type: SCD_TYPE_2, where the overview's feature table allows it.
  • channel: PREVIEW only where the docs examples set it: SharePoint (GA), Google Drive, OneDrive, Kafka, RabbitMQ.

Important

  • An ingestion pipeline has ingestion_definition, never libraries.
  • Ask before reusing a destination schema: a source table whose name matches an existing destination table fails the update.
  • Two tables with the same name can't share a pipeline, even from different source schemas: set destination_table on one, or split. A renamed table, or several destination catalogs or schemas, make the pipeline API-only.
  • A column selected after the pipeline started isn't backfilled. A full refresh backfills it but clears the table and its state and reloads from the source: SCD type 2 history, rows deleted at the source and data the source no longer keeps are lost. Say so and ask before running one.
  • A table deleted at the source keeps its destination table.

Troubleshooting

Read the pipeline's event log (databricks-pipelines), then the connector's -troubleshoot page.

SymptomCauseFix
DUPLICATE_KEY_VIOLATIONThe key the source reports isn't unique per row (NetSuite's line-level tables report transactionid)Set table_configuration.primary_keys to a key that is unique per row, as the connector's -pipeline page names it (NetSuite: transactionid and line), then full refresh that table (ask first)
A working connection fails authRefresh token revoked or expiredReauthorize the connection in Catalog Explorer
Create fails or the connector isn't foundPreviews toggle off, or channel: PREVIEW missingCheck the page's Beta note and examples
Every update re-reads a whole tableNo cursor (entity tables, formula fields, report snapshots)Expected; Salesforce formula fields can be ingested incrementally (Beta) in a new pipeline, API only: Ingest formula fields incrementally
Too many tablesPer-pipeline cap on -limits (250 where stated, NetSuite 200, Outlook 1)Split the pipeline

Reference Files

Lakeflow Connect: databases

Managed ingestion pipelines from databases into Unity Catalog Delta tables (SaaS apps and Kafka / RabbitMQ topics: databricks-lakeflow-connect-saas).

The docs pages win. Release stage, prerequisites, grants, limits, error text and field names change between releases: read the source's pages below before you build, and the workspace's error before this skill. This skill holds the order of steps, the gates, and the traps the pages don't state.

Is Lakeflow Connect the right tool?

If the source is...UseSkill
A database with a managed connectorLakeflow Connectthis skill
A SaaS app (Salesforce, Workday, ServiceNow, SharePoint, ...) or a Kafka / RabbitMQ topicLakeflow Connectdatabricks-lakeflow-connect-saas
Files on cloud object storageAuto Loaderdatabricks-pipelines
A source to query in place, no copyLakehouse Federationn/a
An app or device that pushes eventsZerobusdatabricks-zerobus-ingest

Modes per source

Pages are https://docs.databricks.com/ingestion/lakeflow-connect/<source>-<page>. For a mode marked "-", look for its page first (<source>-integrated-pipeline for integrated CDC, <source>-pipeline for standard CDC): if it exists, follow it; if it isn't found, tell the user Lakeflow Connect's docs have no page for ; if the docs can't be opened, that they had none when this skill was written. Then offer the modes the row links.

Source (<source>)Integrated CDCStandard CDC (gateway)Query-basedSetup, limits, errors
SQL Server (sql-server)integrated-pipelinepipelineyessource-setup, limits, troubleshoot
MySQL (mysql)integrated-pipelinepipelineyessource-setup, limits, troubleshoot
PostgreSQL (postgresql)-pipelineyessource-setup, limits, troubleshoot
Oracle (oracle)integrated-pipeline-yesintegrated-setup, limits, troubleshoot
Teradata, MariaDB, any Federation foreign catalog--yesquery-based overview, limits

Which mode: CDC without a kind named: ask which (integrated, one pipeline; standard, a gateway that runs and bills continuously), unless the source has one. "Standard", "two-pipeline" or "gateway" → standard. Every change or low source load → CDC (hard deletes with query-based: its limits page); otherwise, or when CDC can't be turned on at the source → query-based. A mode marked "-" with no page found (or the docs not reachable): tell the user as in Modes per source, offer the modes it has, and don't author the requested one, even to try it. Never switch to standard silently: its gateway runs and bills continuously.

What marks each mode in the pipeline

Field names and stages: databricks bundle schema (pipelines.IngestionPipelineDefinition). An ingestion pipeline has ingestion_definition, never libraries; CREATE TABLE ... FROM CONNECTION is Federation, not ingestion.

ModePipelinesMarking fields
Integrated CDConeconnection_name + connector_type: CDC; data_staging_options (catalog, schema); every field the source's page says a create request must include (Oracle and MySQL: channel: PREVIEW)
Standard CDCgateway + ingestiongateway: gateway_definition (connection_name, gateway_storage_catalog, _schema, _name); ingestion: ingestion_gateway_id: ${resources.pipelines.<gateway>.id}; source_type and the source's settings where its page's example has them (PostgreSQL: source_type: POSTGRESQL, source_configurations[].catalog.postgres.slot_config)
Query-based, connectiononeconnection_name, no connector_type; per table table_configuration.query_based_connector_config.cursor_columns (a list), primary_keys, scd_type
Query-based, foreign catalogoneingest_from_uc_foreign_catalog: true, no connection_name; source_catalog = the foreign catalog; primary_keys required unless APPEND_ONLY

Every object: table with source_catalog / source_schema / source_table, destination_catalog / destination_schema (destination_table only to rename), or schema for a whole source schema. Per source: sources.md.

Compute

Before creating, find what each pipeline would run on with serverless unset: the mode's pipeline page, and pipelines of the same mode already in this workspace (databricks pipelines list-pipelines; update.config.serverless of their latest update in databricks pipelines get-update <id> <update-id> -o json: true serverless, false classic). Compare that with the compute the user asked for: set serverless (and a clusters block for classic, as the mode's page shows) only when they differ, otherwise leave it unset. After the first update, read the same field to confirm. If the page says a mode or one of its pipelines (a gateway, an ingestion pipeline) can't run on the compute asked for, say so and offer what the page allows. Details and the two compute failures: modes.md.

Before creating anything

Ask for every missing input in one message (source, connection, tables, destination catalog and schema, mode, compute, schedule), list what you'll create, and wait for one "yes"; then run the setup without asking again. Never create a stand-in source without the user's go-ahead. Check:

  1. Target catalog and grants: as the mode's pipeline page lists.
  2. A UC CONNECTION with the source credentials: password as secret('<scope>', '<key>') (CREATE CONNECTION: use the secret function for any credential). After a rotation, run ALTER CONNECTION <name> OPTIONS (..., password secret('<scope>', '<key>')) with the options from the CREATE (DESCRIBE CONNECTION lists only some), then test the connection with a read. Write a request body with a password to a file only you can read, never on a command line.
  3. Network from the compute that reads the source (the integrated pipeline, the gateway, or the query-based pipeline, on the compute chosen above).
  4. Source setup from the source's source-setup page, and its traps in sources.md. Changes to the customer's database (CDC on a table, a slot): give the SQL to the user or their DBA; ask first.
  5. Query-based: the pre-flight checks (key unique, cursor bumped on update, no NULL cursors). Report, and let the user skip them for a snapshot or test.
  6. Names: ask. Demos and tests without names: one <base>_<epoch> for pipeline, schema, job and bundle (standard CDC: <base>_gw_<epoch> and <base>_ap_<epoch>; staging <base>_<epoch>_staging). mode: development prefixes names, so slot and schema names taken from them stop matching.

Run, schedule, clean up

  • databricks bundle deploy, then databricks bundle run <key> --no-wait (or pipelines start-update <id>), then get-update until COMPLETED or FAILED; integrated CDC updates run long by design (smart closure).
  • Schedule with a Jobs pipeline_task; whether the mode can also run continuously, the mode's pipeline page says.
  • After the first load and every full refresh, compare source and destination row counts; a COMPLETED status or low lag isn't proof rows landed. Rows per table per run: rows-per-run.md.
  • Before a full refresh, read what the mode's page says it does to the table and to change-feed readers; warn and ask first.
  • Clean up only with the user's OK: list the pipelines, schemas and PostgreSQL slot you'd remove, and what deleting a pipeline does to its tables (its page). Give the slot SQL to the user or their DBA. Don't stop a gateway the user keeps.

When something fails

Read the full message (CLI error, deploy output, ERROR/WARN events with details), then the source's troubleshoot page, then troubleshooting.md and sources.md. Then decide: a setting to change (take the value the message names), a missing grant (report the privilege and who grants it), a feature not enabled ("Ingestion source '' is not supported in your workspace": report it, offer the modes that work), or a source-side fix. Error names, messages and behaviors in this skill that no docs page lists were seen in test runs: match on the message too, and prefer the docs page where they differ. Fix that one thing and retry the same step; offer bundle destroy for what a failed deploy left.

Reference files

FileWhen
modes.mdIntegrated CDC, standard CDC, query-based: how each behaves, one example each, compute failures
sources.mdSQL Server, PostgreSQL, MySQL, Oracle: names per source, prerequisite traps, errors from runs
troubleshooting.mdEvent log, cross-pipeline errors, table limits, runtime per update
rows-per-run.mdRows per table per run (scripts/rows_per_run.sql)

Related skills

databricks-pipelines (runtime, event log), databricks-dabs (bundles), databricks-jobs (pipeline_task), databricks-unity-catalog (catalogs, connections), databricks-lakebase (Lakebase projects and roles), databricks-zerobus-ingest (push).

Databricks Lakehouse MCP

The hosted system.ai.lakehouse service provides notebook tools and the detailed instructions for using them. This local skill handles discovery and connection; load operational guidance from the connected service for the user's task.

Connect or recover

First use the client's tool discovery to look for configured or deferred Lakehouse MCP tools. If they are available, proceed to skill retrieval. Otherwise, use the setup instructions shown for system.ai.lakehouse in the user's Databricks workspace to configure and authenticate the connection in Claude Code or Codex. Use the service's current setup instructions for connection details; do not guess an endpoint or installer command.

If system.ai.lakehouse or its setup instructions are unavailable, or preview access is denied, direct the user to their Databricks account representative to establish availability and request access. Installing this skill does not grant MCP access. Missing tools alone do not show whether the connection is unconfigured or access is unavailable.

For an authentication error, follow the client's sign-in recovery. For an access denial, use the relevant access-request path. For a network or service error, guide retry or diagnosis. If the cause is unclear, keep it unknown. After explaining a blocker, avoid repeating the same setup prompt on later turns; continue through an existing supported path when appropriate for the request.

Use databricks-core when CLI authentication or profile selection is needed.

Retrieve the task guidance

  1. Discover the connected server's advertised skill tools. Call list_skills(query="<short task keyword>") to search skill IDs, names, and descriptions by case-insensitive substring. Omit query to browse the available guides, including when a search returns no matches. Follow any returned page_token to see more results.
  2. Load the relevant guide with load_skill(skill="<returned skill id>"). This returns the full SKILL.md instructions and supporting-file metadata.
  3. Read the supporting files needed for the task with read_skill_file(skill="<returned skill id>", path="<returned file path>"). Use the exact paths returned by load_skill or list_skill_files. To browse more files, call list_skill_files(skill="<returned skill id>"); an optional prefix, such as "references/", narrows the list. Follow any returned next-call instructions to continue browsing with the same filters and page_token.
  4. Follow the guide and current tool descriptions to perform the task. Load more guides when the task requires them, and reuse applicable guidance while it remains in context. Refresh the relevant guidance after reconnecting or detecting a change in the service's capabilities.

Skill retrieval is a useful connection check; no separate healthcheck is needed. It verifies access to that read, not permission to edit a notebook or readiness to run compute. Handle those checks and failures through the relevant operations. If the server does not advertise skill retrieval, report the missing capability and use the service's current setup/recovery guidance.

Unity Catalog Metric Views

Define reusable, governed business metrics in YAML that separate measure definitions from dimension groupings for flexible querying.

When to Use

Use this skill when:

  • Defining standardized business metrics (revenue, order counts, conversion rates)
  • Building KPI layers shared across dashboards, Genie, and SQL queries
  • Creating metrics with complex aggregations (ratios, distinct counts, filtered measures)
  • Defining window measures (moving averages, running totals, period-over-period, YTD)
  • Modeling star or snowflake schemas with joins in metric definitions
  • Enabling materialization for pre-computed metric aggregations

First: can this even be a metric view? (decision gate)

Before authoring, confirm the measure re-aggregates at query time. Additive SUM/COUNT, a COUNT(DISTINCT) that recomputes per cell, MEASURE() ratios (divide last), and supported window measures (trailing / cumulative / period-over-period — Pattern 8) all re-aggregate correctly and are valid metric-view measures; this list is not exhaustive. The gate below is about disqualifying shapes — if any of the following hold, it is not a metric view: author a governed Unity Catalog SQL function instead (build it with databricks-dbsql) and expose it to Genie as a trusted registered SQL function (Design Priorities surface #11), rather than forcing it into a view or a pre-aggregated helper view:

  • Non-additive at the queried grain — summing it across a dimension double-counts. Distinct-entity counts are the classic trap: an entity appearing under two attribute values is one entity, not two, so a per-cell distinct count must not be summed across an arbitrary multi-attribute cross.
  • Computed per-entity, then aggregated, where the per-entity step isn't a sum/count/ratio-of-sums — e.g. a per-entity min/max across two periods, or a threshold on a per-entity distinct count.
  • Selection-dependent — the result redistributes with the user's filter (an "all others" bucket that shifts every cell).
  • Iterative — repeats until convergence (e.g. panel reweighting).
  • Returns a bundle of tables, not a single measure (e.g. a source×destination matrix plus per-row nets).

A measure can be mostly a metric view with one part hoisted into a function (penetration = additive buyers ÷ base, divide last — but a short-window correction, or an arbitrary multi-attribute cross / per-entity threshold, goes in a function). Pre-computing the awkward step into a base/helper view is only a fix for a fixed definition — a performance choice, settled once for all slices. If the measure must respond to the user's runtime selection in Genie (arbitrary cross, a parameter, or a shifting denominator), it is computed on the fly and cannot be pre-computed — it needs a parameterized governed SQL function, not a pre-aggregated view. Full rule + two worked examples (penetration, brand-switching gains & loss): metric-view-advisor.md §When a metric view is not the right tool. Before benchmarking Genie's routing to a function that reproduces an external methodology, validate it to N-decimal parity against its reference — see the databricks-genie-agents skill.

Prerequisites

  • Databricks Runtime 17.2+ (for YAML version 1.1); 17.3+ for semantic metadata (synonyms / display_name / format)
  • SQL warehouse with CAN USE permissions
  • SELECT on source tables, CREATE TABLE + USE SCHEMA in the target schema

Metric View Lifecycle

TaskReferenceLoad when
Createmetric-view-advisor.mdAny creation task — the advisor handles the full workflow (profile schema, analyze sources, suggest, deploy). Load create-patterns.md alongside as the YAML spec and pattern reference.
YAML spec / patternscreate-patterns.mdPatterns 1–12, full YAML field reference, formatting gotchas, deployment errors, quick reference. Companion to the advisor; also load directly for pattern lookup.
Queryquery-patterns.mdWriting SQL against a metric view — MEASURE() basics, filters, join rollups, window measures, Rules 1–3.
Genie integrationmetric-view-advisor.md §Genie Design RulesOne-fact-source rule, base views, domain organization, naming. Agent metadata fields (comment, synonyms, display_name, format) are in create-patterns.md §YAML Field Reference.

Typical flow: advisor → create → query/validate → Genie integration (if adding to a Genie Agent).

Source-controlled deployment with Declarative Automation Bundles

To source-control a metric view, commit its complete SQL definition and execute it through a bundle-managed SQL job. DABs do not have a native metric-view resource, but a bundle-managed SQL job can apply a committed definition:

# databricks.yml
bundle:
  name: orders_metrics

variables:
  catalog: { default: main }
  schema:  { default: default }
  warehouse_id: { default: "" }

resources:
  jobs:
    deploy_orders_metrics:
      name: deploy_orders_metrics
      parameters:
        - name: catalog
          default: ${var.catalog}
        - name: schema
          default: ${var.schema}
      tasks:
        - task_key: create_metric_view
          sql_task:
            warehouse_id: ${var.warehouse_id}
            file:
              path: ../src/orders_metrics.metric_view.sql

Deploy and run:

databricks bundle deploy --target <TARGET> --profile <PROFILE>
databricks bundle run deploy_orders_metrics --target <TARGET> --profile <PROFILE>

See the official metric view bundle example.

Related Skills

Resources

ML Training on Databricks

FIRST: Use the parent databricks-core skill for CLI basics, authentication, and profile selection.

Train with MLflow → register to Unity Catalog → consume the same artifact as either a batch Spark UDF over Delta or (when low-latency is required) a real-time serving endpoint.

Always train on Databricks (serverless job or notebook), never in the local Python process the agent is running in. Local training has no access to the silver tables, no MLflow tracking server, no UC registry path, and dies if the chat session drops — submit databricks jobs submit --no-wait (see "Train + deploy as a serverless job" below). Only fall back to local execution if the user explicitly asks for it.

If you need to deploy a real time model serving endpoint after the model is registered (creating endpoints, traffic config, version-swapping, querying, Foundation Model API endpoints), see databricks-model-serving.

ConsumptionWhenHow
Batch UDFDashboards, daily/hourly scores, predictions read by Genie/Dashboards or an app (often synced to a Lakebase table)mlflow.pyfunc.spark_udf(...) → INSERT INTO gold_predictions. If the model was logged with fe.log_model(training_set=...), use fe.score_batch() instead — see the Feature Engineering section below.
Real-time endpointScore on a user action (fraud at authorization, rec at page load) — sub-100msmlflow.deployments.get_deploy_client() (classical) / agents.deploy() (agents). Endpoint lifecycle: see databricks-model-serving.

Default Canonical flow

silver_<features>  +  silver_<labels>
        ▼
   notebook (as a serverless job):
   ├── train with mlflow.autolog (XGBoost / sklearn / etc.)
   ├── mlflow.register_model → UC: {catalog}.{schema}.{model}
   ├── set_registered_model_alias(name, "prod", version)
   └── spark_udf(@prod) over latest features → MERGE into gold_predictions
        ▼
gold_<entity>_predictions   ◄── dashboards, apps, Genie read this

Feature-store-backed models diverge here. If training used fe.log_model(training_set=...), replace spark_udf(@prod) with fe.score_batch(model_uri, df=<keys_only>) — it auto-joins features via the model's registered feature lineage. See the Feature Engineering section below.

One notebook, one artifact. Re-running = retraining. Gold is where truth lives — read paths never call the model directly. Keep label-window logic (failure occurred within 7 days) in the notebook during dev; once stable, promote to a silver materialized view in SDP.


Train and register (the 90% case)

mlflow.autolog() captures params, metrics, code, and the model artifact for every run; registered_model_name=... auto-registers the best run to UC (auto-incremented version). Wrap training with Optuna so each trial is a child run and the best one is what gets registered.

Always mlflow.set_registry_uri("databricks-uc") — without it, models land in the deprecated workspace registry. The experiment's parent folder must exist — set_experiment does NOT auto-create it (fails with NOT_FOUND: Parent directory does not exist). Pre-create it once with databricks workspace mkdirs before the job runs.

# Once per project — create the parent folder for the MLflow experiment.
databricks workspace mkdirs /Users/me@example.com/turbine_project

Use the Databricks notebook source format (# Databricks notebook source header, # COMMAND ---------- separators, # MAGIC %md/%sql magics for markdown/SQL cells):

# Databricks notebook source
# MAGIC %md
# MAGIC # Turbine failure prediction
# MAGIC
# MAGIC Train an XGBoost classifier on engineered turbine telemetry features.
# MAGIC ## Data exploration

# COMMAND ----------

# (basic data exploration — class balance, schema sanity, etc.)

# COMMAND ----------
# MAGIC %md
# MAGIC ## Training the model

# COMMAND ----------

import mlflow, mlflow.xgboost, optuna
from mlflow.tracking import MlflowClient
from xgboost import XGBClassifier
from sklearn.metrics import roc_auc_score

mlflow.set_registry_uri("databricks-uc")
mlflow.set_experiment("/Users/me@example.com/turbine_project/mlflow_experiment")

CATALOG, SCHEMA, NAME = "ai_demo_gen", "wind_farm", "turbine_failure"
FULL_NAME = f"{CATALOG}.{SCHEMA}.{NAME}"

# Autolog WITHOUT registered_model_name — otherwise every Optuna trial registers a new UC
# version, and a max-by-version pick lands on the last trial to finish, not the best one.
mlflow.xgboost.autolog(log_input_examples=True)

# For imbalanced labels: stratify the split, set scale_pos_weight = neg/pos.
def objective(trial):
    params = {
        "n_estimators":  trial.suggest_int("n_estimators", 100, 400),
        "max_depth":     trial.suggest_int("max_depth", 3, 10),
        "learning_rate": trial.suggest_float("learning_rate", 0.01, 0.3, log=True),
    }
    with mlflow.start_run(nested=True):
        m = XGBClassifier(**params).fit(X_train, y_train)
        return roc_auc_score(y_test, m.predict_proba(X_test)[:, 1])

with mlflow.start_run(run_name="hpo") as parent:
    study = optuna.create_study(direction="maximize")
    study.optimize(objective, n_trials=20)

# COMMAND ----------
# MAGIC %md
# MAGIC ## Retrain best params and register

# COMMAND ----------
# Retrain on the winning trial's params explicitly, then register that single model.
with mlflow.start_run(run_name="best"):
    best = XGBClassifier(**study.best_params).fit(X_train, y_train)
    mlflow.log_metric("val_auc", study.best_value)
    info = mlflow.xgboost.log_model(best, name="model", registered_model_name=FULL_NAME)

# Stages are deprecated — UC uses movable aliases. Repoint @prod at the version we just registered.
client = MlflowClient(registry_uri="databricks-uc")
client.set_registered_model_alias(FULL_NAME, "prod", info.registered_model_version)

Framework autolog: mlflow.{sklearn,xgboost,lightgbm,pytorch,tensorflow,spark}.autolog().

Aliases, not stages: UC dropped Staging/Production. Use movable @prod/@challenger; load with models:/{full_name}@prod. Promoting a new version is one set_registered_model_alias call.


Consume: batch scoring over Delta

The cheap, default path for models NOT backed by feature tables. Load the registered model as a Spark UDF and score a Delta table; write predictions to a gold table that downstream consumers read. For feature-store-backed models (logged with fe.log_model(training_set=...)), skip spark_udf entirely and use fe.score_batch() — see the Feature Engineering section.

# COMMAND ----------
# MAGIC %md
# MAGIC ## Score and save to a gold predictions table

# COMMAND ----------
import mlflow
from pyspark.sql import functions as F

# env_manager rules:
#   "local"     → same runtime as training (same notebook/job). Fastest, default in dev/demo.
#   "virtualenv"→ different runtime than training; rebuilds the model's env.
#   "uv"        → same as virtualenv but faster (MLflow ≥ 2.22).
predict = mlflow.pyfunc.spark_udf(
    spark,
    model_uri=f"models:/{FULL_NAME}@prod",
    env_manager="local",
)

features = spark.table(f"{CATALOG}.{SCHEMA}.silver_turbine_features_latest")
feature_cols = [c for c in features.columns if c != "turbine_id"]   # exclude the join key
scored = features.withColumn("risk_score", predict(*[features[c] for c in feature_cols]))

# Overwrite-per-run pattern for "latest score per entity":
scored.select("turbine_id", "risk_score", F.current_timestamp().alias("scored_at")) \
    .write.mode("overwrite").saveAsTable(f"{CATALOG}.{SCHEMA}.gold_turbine_predictions")

For incremental scoring with history, MERGE into the predictions table instead of overwrite.


Real-time serving (when required)

After registering a model to UC, deploy it behind a Model Serving endpoint. The dev-side call is mlflow.deployments.get_deploy_client("databricks").create_endpoint(...) for classical ML or agents.deploy(...) for ResponsesAgents. First deploy is ~5 min for classical ML.

For endpoint create / update / version-swap, traffic config, AI Gateway, querying, the state.ready + state.config_update two-field readiness check, and Foundation Model API endpoints, see databricks-model-serving.


Train + deploy as a serverless job

Training notebooks run a few minutes (Optuna + UC register; endpoint warmup adds 5–15 min if you also deploy). Submit as a serverless one-time run so the CLI doesn't block. The notebook ends with dbutils.notebook.exit(json.dumps({...})) so the structured result (model_version, val_auc, endpoint_name) reaches .notebook_output.result.

# 1. Upload the training notebook
databricks workspace import /Workspace/Users/me@example.com/turbine_project/train \
  --file ./train_notebook.py --format SOURCE --language PYTHON --overwrite

# 2. Submit as serverless one-time run (returns {"run_id": N} immediately with --no-wait)
RUN_ID=$(databricks jobs submit --no-wait --json '{
  "run_name": "turbine-train-and-deploy",
  "tasks": [{
    "task_key": "train",
    "notebook_task": {"notebook_path": "/Workspace/Users/me@example.com/turbine_project/train"},
    "environment_key": "ml_env"
  }],
  "environments": [{
    "environment_key": "ml_env",
    "spec": {
      "client": "4",
      "dependencies": ["mlflow==3.1.0", "xgboost==2.1.3", "optuna==4.1.0", "scikit-learn==1.5.2"]
    }
  }]
}' | jq -r .run_id)

# 3. Poll until a terminal life_cycle_state.
for _ in $(seq 60); do
  STATE=$(databricks jobs get-run "$RUN_ID" | jq -r '.state.life_cycle_state // "UNKNOWN"')
  echo "$(date +%H:%M:%S) $STATE"
  [[ "$STATE" =~ ^(TERMINATED|SKIPPED|INTERNAL_ERROR)$ ]] && break
  sleep 30
done
[[ "$STATE" =~ ^(TERMINATED|SKIPPED|INTERNAL_ERROR)$ ]] || { databricks jobs cancel-run "$RUN_ID"; exit 1; }

# life_cycle_state TERMINATED only means "the run ended" — check result_state.
RESULT=$(databricks jobs get-run "$RUN_ID" | jq -r '.state.result_state // "UNKNOWN"')
echo "result_state=$RESULT"
[[ "$RESULT" == "SUCCESS" ]] || { echo "Run did not succeed"; exit 1; }

# 4. Pull structured output via the TASK run_id (NOT the submit run_id).
TASK_RUN_ID=$(databricks jobs get-run "$RUN_ID" | jq -r '.tasks[0].run_id')
databricks jobs get-run-output "$TASK_RUN_ID" | jq '.notebook_output.result'
# → '{"model_version":"3","val_auc":0.91,"rows_scored":124,"endpoint":"turbine-risk-endpoint"}'

Common jobs submit traps to be aware of: environments[].spec.client: "4" is required on serverless notebook tasks; use the TASK run_id (tasks[0].run_id) — NOT the submit run_id — for get-run-output; print() is unreliable on serverless one-time runs (use dbutils.notebook.exit(json.dumps(...))); jobs submit rejects tags. For the broader databricks-jobs skill, see databricks-jobs.


Custom pyfunc

When sklearn/XGBoost autolog isn't enough — custom preprocessing, multiple sub-models, external API calls, ensemble logic. See references/custom-pyfunc.md for a full worked example. Two non-obvious things:

  • python_model="path/to/file.py" (file path, not class instance) + mlflow.models.set_model(MyModel()) at the end of that file. This is the "Models from Code" pattern — the file is logged verbatim, no pickling of the class.
  • mlflow.models.predict(model_uri=..., input_data=..., env_manager="uv") before deploying. Catches missing deps before the endpoint does.

Custom GenAI agents

Hand-rolled ResponsesAgent (LangGraph + UC Function tools + Vector Search retrieval) — see references/genai-agents.md.

Prefer no-code authoring via databricks-genie-agents (tables + documents in UC volumes; replaces the legacy Knowledge Assistant / Supervisor Agent) unless the user explicitly needs a custom LangGraph agent.


Feature Engineering: Feature Store & Feature Views

When to reach for Feature Engineering (either flavor) instead of a plain Delta table: when the same feature must be computed identically at training and serving time (no training/serving skew), the feature is time-dependent and needs point-in-time joins against labels (no future leakage), the feature needs to be served with <10ms latency via an online store, or the feature is shared across models with lineage tracked in UC. If none apply, a plain UC table is enough — the sections below can be skipped.

Default: don't use Feature Engineering unless one of the reasons above clearly applies or the user explicitly asked for it. It adds build time and complexity (an extra Delta table layer for FeatureLookup; a materialization pipeline for Feature Views). If you're unsure, use plain UC tables and add Feature Engineering later when a concrete need surfaces.

Two flavors, one train/score path (create_training_set → fe.log_model → fe.score_batch):

  • FeatureLookup API — you own the feature tables (compute, write, refresh); bind them to a training set via FeatureLookup + FeatureEngineeringClient. Reach for it when the features already exist as Delta tables or you want direct control over how they are computed. See references/feature-store.md.
  • Feature Views (declarative, GA, databricks-feature-engineering>=0.16.0) — Databricks owns compute, materialization (Delta offline + Lakebase online), and refresh from a spec you declare (create_feature over DeltaTableSource; RollingWindow/SlidingWindow/TumblingWindow; create_stream for Kafka features). Reach for it when the feature is a formula you want Databricks to keep fresh. See references/feature-views.md.

Gotchas (the ones that cost time)

TrapFix
Model lands in workspace registry, not UCmlflow.set_registry_uri("databricks-uc") before logging
Endpoint returns PERMISSION_DENIED at first queryPass resources=[...] to log_model (covers UC functions, VS indexes, other endpoints, Lakebase) — see references/genai-agents.md#resources-that-need-passthrough-auth for the full list
Used transition_model_version_stageStages are deprecated in UC. Use client.set_registered_model_alias(name, "prod", version)
spark_udf rebuilds a virtualenv on every callPass env_manager="local" when training+scoring share a runtime
pip_requirements mismatch crashes endpoint at loadPin exact versions; or pull live with f"mlflow=={get_distribution('mlflow').version}"
agents.deploy() produced a weirdly-named endpointPass endpoint_name=... explicitly. Auto-derived name is agents_<catalog>-<schema>-<model>

Endpoint-lifecycle gotchas (readiness two-state, version-swap, Serving-UI SP filter) live in databricks-model-serving.


Reference files

FileContents
references/custom-pyfunc.mdSingle end-to-end custom pyfunc example: artifacts, signature, code_paths, log → register → deploy → query.
references/genai-agents.mdCustom LangGraph ResponsesAgent with UC Function + Vector Search tools. create_text_output_item gotcha and the resources=[...] passthrough-auth list. For no-code agents prefer databricks-genie-agents.
references/feature-store.mdFeature Engineering in Unity Catalog (standard FeatureLookup API): FeatureEngineeringClient, create_table with timeseries_column, FeatureLookup with point-in-time joins, fe.log_model with lineage, fe.score_batch (replaces spark_udf for FE models), and Lakebase online store via publish_table. Requires databricks-feature-engineering>=0.16.0.
references/feature-views.mdFeature Views (declarative Feature Engineering, GA): create_feature over DeltaTableSource, RollingWindow/SlidingWindow/TumblingWindow aggregations, materialize_features (offline + online), streaming features off Kafka via create_stream, point-in-time training sets, and the Feature Serving Endpoint. Requires databricks-feature-engineering>=0.16.0.

Related skills

Databricks MLflow evaluation

Read databricks-core for authentication and workspace discovery. For tracing, trace retrieval, observability, and trace-based debugging, use the separate databricks-mlflow-tracing skill. Verify tracing when the selected evaluation mode or scorers require it; reuse precomputed outputs or existing traces without rerunning the agent.

The guides below are references, not separately installed skills. When a guide names an upstream MLflow skill, use the matching guide here or the tracing skill.

RequestRead this guide
Define quality criteria or scorersScorer design
Create a dataset and evaluate an agentMLflow evaluation
Change agent behavior and verify improvementImprove an agent

For MLflow API details, consult the existing databricks-docs skill and https://mlflow.org/docs/latest/llms.txt.

For Databricks judge models, prefer databricks:/system.ai.<model> with a mlflow[databricks]>=3.16.0. A 404 commonly indicates an older client that does not understand the system model URI; a 403 on a legacy endpoint name can mean legacy serving endpoints are disabled. Verify the installed client and available system.ai model before changing credentials or permissions.

For a new evaluation, establish a baseline with a fixed dataset and scorers, change one thing, then run the same evaluation again. Use databricks-mlflow-tracing first if traces are missing or a failure needs a trace-grounded diagnosis.

Databricks MLflow Tracing

Use this skill for MLflow observability and tracing workflows on Databricks. Read databricks-core first for CLI authentication, profile selection, and workspace discovery. On Databricks, reuse the existing experiment ID or choose an absolute workspace path such as /Users/<email>/my-experiment or /Shared/my-experiment; relative names in the bundled onboarding examples are for non-Databricks tracking servers.

Read the matching guide directly:

RequestRead this guide
Choose a use case or get started with MLflowMLflow onboarding
Instrument an app or agent and verify tracingMLflow tracing
Retrieve traces by ID, time, status, or metadataRetrieve traces
Inspect one trace or explain an agent failureAnalyze a trace
Reconstruct or debug a multi-turn chat sessionAnalyze a chat session
Query token, latency, or quality metrics on a supported OSS MLflow serverQuery MLflow metrics

For agent evaluation, scorer creation, or implementing a fix after diagnosis, use databricks-mlflow-evaluation.

For work that spans multiple guides, use this order:

  1. Determine the use case with MLflow onboarding.
  2. Add or verify tracing with MLflow tracing.
  3. Retrieve traces before trace or chat-session analysis.
  4. Use Query MLflow metrics for aggregate trends.

For Databricks trace storage, read the tracing guide's Databricks reference before claiming that traces use Unity Catalog storage.

For a Unity Catalog trace location, verify the deployed identity rather than a local user. It needs experiment access, CAN USE on the SQL warehouse, USE CATALOG and USE SCHEMA, and explicit MODIFY, SELECT on each generated spans, logs, metrics, and annotations table. ALL PRIVILEGES alone does not satisfy those table grants. Confirm persistence by reading back a fresh trace written under the deployed identity.

Model Serving Endpoints

FIRST: Use the parent databricks-core skill for CLI basics, authentication, and profile selection.

Gateway boundary: This skill owns legacy AI Gateway configuration attached to a Model Serving endpoint, such as serving-endpoints put-ai-gateway. An explicit Unity Gateway / Unity AI Gateway request or any databricks ai-gateway model service, MCP service, or model provider service belongs to databricks-unity-gateway.

Model Serving provides managed endpoints for serving LLMs, custom ML models, and external models as scalable REST APIs. Endpoints are identified by name (unique per workspace).

Endpoint Types

TypeWhen to UseKey Detail
Pay-per-tokenFoundation Model APIs (Llama, GPT-5, Claude, Gemini, etc.)Uses system.ai.* catalog models, pre-provisioned in every workspace. Discover at runtime — see Foundation Model API endpoints below.
Provisioned throughputDedicated GPU capacityGuaranteed throughput, higher cost
Custom modelYour own MLflow models or containersDeploy any model with an MLflow signature

Endpoint Structure

Serving Endpoint (top-level, identified by NAME)
  ├── Config
  │     ├── Served Entities (model references + scaling config)
  │     └── Traffic Config (routing percentages across entities)
  ├── AI Gateway (rate limits, usage tracking)
  └── State (READY / NOT_READY, config_update status)
  • Served Entities: Each entity references a model (from Unity Catalog or MLflow) with scaling parameters. Get the entity name from served_entities[].name in the get output — needed for build-logs and logs commands.
  • Traffic Config: Routes requests across served entities by percentage (for A/B testing, canary deployments).
  • State: Endpoints transition NOT_READY → READY after creation or config update. Poll via get to check state.ready.

CLI Discovery — ALWAYS Do This First

Do NOT guess command syntax. Discover available commands and their usage dynamically:

# List all serving-endpoints subcommands
databricks serving-endpoints -h

# Get detailed usage for any subcommand (flags, args, JSON fields)
databricks serving-endpoints <subcommand> -h

Run databricks serving-endpoints -h before constructing any command. Run databricks serving-endpoints <subcommand> -h to discover exact flags, positional arguments, and JSON spec fields for that subcommand.

Create an Endpoint

Do NOT list endpoints before creating.

databricks serving-endpoints create \
  --json '{
    "name": "<ENDPOINT_NAME>",
    "served_entities": [{
      "entity_name": "<MODEL_CATALOG_PATH>",
      "entity_version": "<VERSION>",
      "min_provisioned_throughput": 0,
      "max_provisioned_throughput": 0,
      "workload_size": "Small",
      "scale_to_zero_enabled": true
    }],
    "traffic_config": {
      "routes": [{
        "served_entity_name": "<ENTITY_NAME>",
        "traffic_percentage": 100
      }]
    }
  }' --profile <PROFILE>
  • Discover available Foundation Models: see Foundation Model API endpoints below for the runtime-list snippet and default-picking rules. You can also check the system.ai catalog in Unity Catalog, or run databricks serving-endpoints list --profile <PROFILE> to see what's deployed in the workspace. Use databricks serving-endpoints get-open-api <ENDPOINT_NAME> --profile <PROFILE> to inspect a specific endpoint's API schema.
  • Long-running operation; the CLI waits for completion by default. Use --no-wait to return immediately, then poll:
    databricks serving-endpoints get <ENDPOINT_NAME> --profile <PROFILE>
    # Check: state.ready == "READY"
    
  • For provisioned throughput or custom model endpoints, run databricks serving-endpoints create -h to discover the required JSON fields for your endpoint type.

MLflow Deployments client (Python alternative)

mlflow.deployments.get_deploy_client("databricks").create_endpoint(name=..., config={...}) takes the same JSON shape as the CLI. Two gotchas:

  • tags= is a top-level kwarg, NOT a field inside config. Same [{key, value}] shape as serving-endpoints patch --add-tags.
  • traffic_config.routes[].served_model_name = "<model>-<version>" (e.g. "turbine_failure-3"). The API auto-derives this from the entity, but you reference the exact string in traffic_config — get the format wrong and the route silently doesn't match.

Zero-downtime version swap

To roll an endpoint to a new model version: repoint the alias and call update_endpoint with the new served_entities + matching traffic_config. Missing either half is the common bug — alias-only doesn't update the endpoint; update_endpoint-only leaves the alias pointing at the old version.

from mlflow.tracking import MlflowClient
from mlflow.deployments import get_deploy_client

registry = MlflowClient(registry_uri="databricks-uc")
deploy   = get_deploy_client("databricks")

registry.set_registered_model_alias(FULL_NAME, "prod", new_version)
deploy.update_endpoint(endpoint=ENDPOINT_NAME, config={
    "served_entities": [{"entity_name": FULL_NAME, "entity_version": new_version,
                         "workload_size": "Small", "scale_to_zero_enabled": True}],
    "traffic_config": {"routes": [
        {"served_model_name": f"{NAME}-{new_version}", "traffic_percentage": 100}
    ]},
})

The CLI equivalent is databricks serving-endpoints update-config <NAME> --json '...'. Either way, poll both state.ready and state.config_update afterward — see Endpoint Readiness below.

Endpoint Readiness

After create or update-config, the endpoint provisions compute and loads the model. Do not query the endpoint until it is ready. Two state fields matter and they mean different things:

  • state.ready — READY once the endpoint has any working config. Stays READY during a version swap.
  • state.config_update — NOT_UPDATING once the current config update finishes; IN_PROGRESS during a version swap.

A loop watching only state.ready will say "ready" mid version-swap while the old version is still serving. Poll both:

databricks serving-endpoints get <ENDPOINT_NAME> --profile <PROFILE> \
  | jq '{ready: .state.ready, config_update: .state.config_update}'
# Fully ready when ready == "READY" AND config_update == "NOT_UPDATING"

Provisioning may take several minutes. Provisioned throughput endpoints take the longest (GPU allocation). Queries to endpoints that are not yet READY return 404 or 503.

Query an Endpoint

Chat / agent endpoints use the messages array:

databricks serving-endpoints query <ENDPOINT_NAME> \
  --json '{"messages": [{"role": "user", "content": "Hello"}]}' --profile <PROFILE>

Classical-ML endpoints use dataframe_records (one record per row):

databricks serving-endpoints query <ENDPOINT_NAME> \
  --json '{"dataframe_records": [{"vibration": 0.42, "rpm": 18.3, "temp_c": 71.2}]}'
  • Use --stream for streaming responses on chat endpoints.
  • For embeddings or other custom schemas: use get-open-api <ENDPOINT_NAME> first to discover the request/response shape.

Get Endpoint Schema (OpenAPI)

Returns the OpenAPI 3.1 JSON schema describing what each served model accepts and returns. Use this to understand an endpoint's input/output format before querying it.

databricks serving-endpoints get-open-api <ENDPOINT_NAME> --profile <PROFILE>

The schema shows paths per served model (e.g., /served-models/<model-name>/invocations) with full request/response definitions including parameter types, enums, and nullable fields.

Other Commands

Run databricks serving-endpoints <subcommand> -h for usage details.

TaskCommandNotes
List all endpointslist
Get endpoint detailsget <NAME>Shows state, config, served entities
Delete endpointdelete <NAME>
Update served entities or trafficupdate-config <NAME> --json '...'Zero-downtime: old config serves until new is ready
Rate limits & usage trackingput-ai-gateway <NAME> --json '...'
Update tagspatch <NAME> --json '...'
Build logsbuild-logs <NAME> <SERVED_MODEL>Get SERVED_MODEL from get output: served_entities[].name
Runtime logslogs <NAME> <SERVED_MODEL>
Metrics (Prometheus format)export-metrics <NAME>
Permissionsget-permissions <ENDPOINT_ID>⚠️ Uses endpoint ID (hex string), not name. Find ID via get.

What's Next

Integrate with a Databricks App

After creating a serving endpoint, wire it into a Databricks App.

Step 1 — Check if the serving plugin is available in the AppKit template:

databricks apps manifest --profile <PROFILE>

If the output includes a serving plugin, scaffold with:

databricks apps init --name <APP_NAME> \
  --features serving \
  --set "serving.serving-endpoint.name=<ENDPOINT_NAME>" \
  --run none --profile <PROFILE>

Step 2 — If no serving plugin, add the endpoint resource manually to an existing app's databricks.yml:

resources:
  apps:
    my_app:
      resources:
        - name: my-model-endpoint
          serving_endpoint:
            name: <ENDPOINT_NAME>
            permission: CAN_QUERY

And inject the endpoint name as an environment variable in app.yaml:

env:
  - name: SERVING_ENDPOINT
    valueFrom: serving-endpoint

Then wire the endpoint into your app via the serving() plugin or a custom route in onPluginsReady. For the full app integration pattern, use the databricks-apps skill and read the Model Serving Guide.

Develop & deploy new models

This skill is ops-focused (manage existing endpoints). For the dev-side flow — training, MLflow tracking, UC registration, custom PyFunc authoring, and hand-rolled ResponsesAgent code — see databricks-ml-training (experimental).

Foundation Model API endpoints

Pay-per-token, pre-provisioned in every workspace. New models land regularly and a static skill list goes stale fast — always list at runtime instead of hard-coding names. Filter by the databricks- name prefix AND by the served entity being in system.ai.* (other endpoints like databricks-app-template-serving share the prefix but aren't FM API endpoints). Use Databricks Unity Gateway (databricks-unity-gateway) to manage access to foundation model endpoints. Do not use databricks-model-serving legacy AI Gateway.

# FM API endpoints in this workspace, grouped by task (chat / embeddings / etc.)
databricks serving-endpoints list \
  | jq -r '.[]
      | select(.name | startswith("databricks-"))
      | select((.config.served_entities[0].entity_name // "") | startswith("system.ai."))
      | "\(.task)\t\(.name)"' \
  | sort

Defaults when the user doesn't specify: pick the highest-numbered Claude Sonnet for agents, the highest-numbered -codex-max for code, databricks-gte-large-en for embeddings — resolve actual names from the live list above.

Off-platform streaming

For apps deployed outside Databricks Apps (Vercel, AWS, standalone Node.js) hitting Databricks legacy AI Gateway with Vercel AI SDK v6, see references/off-platform-streaming.md. Off-platform apps should be updated to hit Databricks Unity Gateway (databricks-unity-gateway). For AppKit-based apps, use the databricks-apps skill's built-in serving plugin instead.

Troubleshooting

ErrorSolution
cannot configure default credentialsUse --profile flag or authenticate first
PERMISSION_DENIEDCheck workspace permissions; for apps, ensure serving_endpoint resource declared with CAN_QUERY
Endpoint stuck in NOT_READYWait up to 30 min for provisioned throughput. Check build logs: build-logs <NAME> <ENTITY_NAME> (get entity name from get output → served_entities[].name)
RESOURCE_DOES_NOT_EXISTVerify endpoint name with list
Query returns 404Endpoint may still be provisioning; check state.ready via get
RATE_LIMIT_EXCEEDED (429)AI Gateway rate limit; check put-ai-gateway config or retry after backoff
Endpoint missing from the Serving UI after deployUI filter defaults to "Owned by me". Deploy jobs run as a service principal, so the endpoint is hidden until you switch to "All". databricks serving-endpoints list always shows it.

Lakeflow Spark Declarative Pipelines Development

FIRST: Use the parent databricks-core skill for CLI basics, authentication, profile selection, and data discovery commands.

SDP, LDP, Lakeflow Declarative Pipelines, and DLT (former name) all refer to the same product. This skill covers the core syntax and rules for writing Spark Declarative Pipelines datasets. For file ingestion and document parsing read file-ingestion; for failed, slow, or flaky updates read pipeline-debugging; for migrating dbt or Informatica projects read migrate-to-sdp.

Decision Tree

Use this tree to choose the dataset type first, then layer on applicable features. Multiple features can apply to the same dataset — e.g., a Streaming Table can use Auto Loader for ingestion, Append Flows for fan-in, and Expectations for data quality.

User request → What kind of output?
├── Intermediate/reusable logic (not persisted) → Temporary View (default)
│   ├── Preprocessing/filtering before Auto CDC → Temporary View feeding CDC flow
│   ├── Shared intermediate streaming logic reused by multiple downstream tables
│   ├── Pipeline-private helper logic (not published to catalog)
│   ├── Expensive intermediate that is reused across refreshes → Private ST/MV (materialized but pipeline-scoped)
│   └── Published to UC for external queries → Persistent View (SQL only)
├── Persisted dataset
│   ├── Source is a periodic complete snapshot that must be diffed by key → Streaming Table + Auto CDC FROM SNAPSHOT
│   ├── Source is streaming/incremental/continuously growing → Streaming Table
│   │   ├── File ingestion (cloud storage, Volumes) → Auto Loader (see [file-ingestion](references/file-ingestion.md))
│   │   ├── Message bus (Kafka, Kinesis, Pub/Sub, Pulsar, Event Hubs) → streaming source read
│   │   ├── Existing streaming/Delta table → streaming read from table
│   │   ├── CDC / upserts / track changes / keep latest per key / SCD Type 1, 2, or bitemporal → Auto CDC (only when this pipeline owns the target; merge/upsert into an existing/external table → ForEachBatch Sink, see Sink branch)
│   │   ├── Source rows carry two timestamps with different semantic meaning — one for when the fact holds in the domain, the other for when the system learned or recorded it (e.g. effective_at + ingested_at, trade date + booking date, observation time + chart-entry time) → Auto CDC STORED AS BITEMPORAL (Beta, DBR 17.3+); read the Auto CDC reference before recommending.
│   │   ├── Multiple sources into one table → Append Flows (NOT union)
│   │   ├── Historical backfill + live stream → one-time Append Flow + regular flow
│   │   ├── Recompute/overwrite a bounded window or partition each run (late-arriving within a known lookback, rolling last-N-days aggregate, partition overwrite, scheduled re-runs) → REPLACE WHERE flow (NOT a Materialized View, plain streaming table, or full recompute)
│   │   ├── Streaming source re-emits all rows for a key and the latest should replace the old per key (re-ingesting mutable files, keyed in-place updates) → REPLACE USING flow (NOT Auto CDC, which is for change events/SCD; NOT REPLACE WHERE, which overwrites a predicate window not a key)
│   │   ├── Windowed aggregation with watermark → stateful streaming
│   │   └── Custom per-key state, timers, timeouts (Python) → transformWithState in a flow (timeMode EventTime or None in triggered pipelines)
│   └── Source is batch/historical/full scan → Materialized View
│       ├── Aggregation/join across full dataset (GROUP BY, SUM, COUNT, etc.)
│       ├── Gold layer aggregation from streaming table → MV with batch read (spark.read / no STREAM)
│       ├── JDBC/Federation/external batch sources
│       └── Small static file load (reference data, no streaming read)
├── Output to a target the pipeline does NOT own (external / pre-existing / managed elsewhere) → Sink (Python only, primary even if the user asked for SQL)
│   ├── External or pre-existing Delta table NOT owned by this pipeline → Sink with format="delta" (streaming tables/MVs are for tables this pipeline owns and manages — including ones it already created and refreshes each run — but they cannot write into a table owned outside the pipeline)
│   ├── Kafka / Event Hubs → Sink with format="kafka" + @dp.append_flow(target="sink_name")
│   ├── Custom destination not natively supported → Sink with custom format
│   ├── Custom merge/upsert into an existing/external table → ForEachBatch Sink, NOT Auto CDC
│   └── Multiple destinations per batch → ForEachBatch Sink
└── Data quality constraints → Expectations (on any dataset type)

Common Traps

  • Changing dataset type (ST↔MV): Read changing-dataset-type BEFORE edits or commands. Create a NEW dataset name and repoint ALL downstream references; never convert in place. Full refresh cannot fix this. The platform's "drop first" message is a manual workaround; offer dropping only as a follow-up. A green dry run does not prove rewiring.
  • SCD Type 2 queries: Use validity columns __START_AT / __END_AT, not invented names. Current rows: __END_AT IS NULL. Point-in-time predicate: __START_AT <= t AND (__END_AT > t OR __END_AT IS NULL); temporal joins also match the key and use the event date as t. Read the Auto CDC reference (Python, SQL) and scd-2-querying.
  • UNRESOLVED_COLUMN / UNRESOLVED_COLUMN.WITH_SUGGESTION: Verify upstream columns. For Auto Loader, verify its schema includes them (e.g., via cloudFiles.schemaHints / schemaHints); read auto-loader-schema.
  • Existing ST column add/drop/rename/widen, DELTA_MERGE_INCOMPATIBLE_DATATYPE or DELTA_FAILED_TO_MERGE_FIELDS: Read streaming-table-schema-evolution before answering; ALTER TABLE is Beta with prerequisites.
  • "Create a table" without a type: Ask whether the source is streaming or batch. Streaming → Streaming Table; batch → Materialized View. Mismatched pairs fail validation.
  • Aggregating a streaming source: For per-window results (hourly, daily) from an append-only source, use a streaming table with a WATERMARK and window(), which bounds state (SQL / Python streaming patterns). Otherwise use a Materialized View with a batch read (no STREAM): streaming tables are append-only and don't recompute aggregates when source rows change.
  • Cannot create streaming table from batch query: The streaming table reads in batch. Use FROM STREAM read_files(...) / FROM STREAM(table).
  • Python-only features (sinks, ForEachBatch sinks, custom data sources, Real-Time Mode): When the user works in SQL, say so and suggest Python for that part.
  • databricks fs on Volumes: The CLI needs the dbfs: prefix, e.g. databricks fs ls dbfs:/Volumes/<catalog>/<schema>/<volume>/<path>.
  • SQL parameters (DBR 17.0+): Read parameters-sql before writing. Use :name for values that vary across updates, not hardcoding or legacy ${name}. Values are STRING: cast at use; identifiers need IDENTIFIER().
  • Recommend one approach. Omit anti-patterns and inferior alternatives; mention alternatives only for viable trade-offs.

Publishing Modes

Pipelines publish to the default catalog/schema in their settings.

  • Use dataset-only names for pipeline reads/definitions unless existing code or the user requests qualification.
  • catalog.schema.table publishes a pipeline-owned dataset elsewhere; no sink needed. External, pre-existing, or separately managed targets require a Python sink, even if the user requested SQL.
  • USE CATALOG / USE SCHEMA changes the defaults for subsequent definitions in the same SQL file.

Comprehensive API Reference

MANDATORY: Before implementing, editing, or suggesting feature code, read its linked reference and these required companion references:

  • Auto Loader, Auto CDC, sinks → also read the streaming-table reference: Auto Loader/CDC target streaming tables; sinks use streaming append flows.
  • Auto Loader → also read the matching format options: CSV, JSON, Avro, Parquet, XML, Text, ORC. For schema inference/evolution, hints, rescued data, partition inference or type widening, read schema.
  • Expectations → also read the target dataset's definition reference for constraint placement.
  • Kafka / Event Hubs sources → also read authentication: Python or SQL (SASL, MSK IAM, Entra ID, GCP OAuth; shaded class names required).
    • Avro with Confluent Schema Registry → read Kafka deserialization, Python only. For SQL pipelines, explain the limitation and recommend switching to Python. Protobuf also works in SQL (from_protobuf, DBR 16.1+; see the SQL deserialization reference).
    • JSON/Avro without Schema Registry → Python or SQL, matching the pipeline language.
  • Real-Time Mode → read reference. Only for explicitly requested operational sub-second latency; requires continuous mode + a real-time update flow.

Feature references

FeaturePythonSQL
Streaming tables; explicit targets; append and one-time backfill flows; skip upstream change commitsPythonSQL
Materialized views; manual/auto liquid clustering, partitioning, table properties, explicit schemas, generated columns, row filters and column masks, private datasetsPythonSQL
Temporary viewsPythonSQL
Persistent UC viewsUnsupportedSQL
Delta/Kafka/Event Hubs/custom sinksPythonUnsupported
ForEachBatch sinksPythonUnsupported
Auto CDC from a streamPythonSQL
Auto CDC from periodic snapshotsPythonSQL
Expectations: warn, drop, fail; individual or multiple constraintsPythonSQL
Identity columnsPythonSQL
Auto LoaderPythonSQL
Kafka/Event Hubs sourcesPythonSQL
Kinesis sourcesPythonSQL
Payload deserialization (JSON/Avro/Protobuf; Avro with Schema Registry is Python only)PythonSQL
  • REPLACE WHERE: overwrite a bounded window/partition each run, such as a rolling lookback. Read Python or SQL.
  • REPLACE USING: streaming keyed replacement of full rows with a required sequencing column; Beta, requires DBR 18.2+. Read Python or SQL.

Topics covering both languages:

  • Streaming patterns (SQL / Python): dedup, windowed aggregations, late data, stream-stream joins, rescued-data quarantine, lag monitoring, transformWithState.
  • SCD Type 2 querying: current state, point-in-time, joining facts with historical dimensions.

Reads and compatibility

  • Pipeline batch reads: spark.read.table("name") / SELECT ... FROM name; streaming reads: spark.readStream.table("name") / SELECT ... FROM STREAM name.
  • Pub/Sub and Pulsar sources: use spark.readStream.format("pubsub") or spark.readStream.format("pulsar"); SQL uses STREAM read_pubsub(...) or STREAM read_pulsar(...).
  • JDBC/Federation are batch sources; Python uses spark.read.format("postgresql") etc. and SQL references the federation catalog.
  • Static files: spark.read.format(...).load(...) / read_files(...) without STREAM.
  • Custom data sources are Python-only: spark.read[Stream].format("custom").
  • Prefer from pyspark import pipelines as dp for new Python. Preserve existing dlt imports. Prefer spark.read.table / spark.readStream.table over the legacy dp.read, dlt.read, and their streaming variants.
  • Never use LIVE. references or legacy CREATE LIVE TABLE / CREATE LIVE VIEW syntax; use current dataset definitions. The one exception is CREATE LIVE VIEW for expectations on a temporary view (see the table below).

Migrating legacy DLT syntax

When the user asks to migrate legacy DLT code to the modern form, read dlt-migration.md before suggesting changes so the conversion is correct (especially around apply_changes → create_auto_cdc_flow semantics and partition_cols → cluster_by). Common mappings:

If you see……migrate to
import dltfrom pyspark import pipelines as dp
@dlt.table(...), @dlt.append_flow(...), @dlt.expect*Same decorator name on dp.* (e.g. @dp.table, @dp.expect_or_drop).
@dlt.view(...) (or @dp.view(...) if present in older code)@dp.temporary_view(...)
dlt.read("name") / dlt.read_stream("name")spark.read.table("name") / spark.readStream.table("name")
dp.read(...) / dp.read_stream(...)Also legacy — use spark.read.table(...) / spark.readStream.table(...).
dlt.apply_changes(...) / dp.apply_changes(...)dp.create_auto_cdc_flow(...). sequence_by accepts a column name (string) or col(...); stored_as_scd_type is 1 or 2 (integer or string).
dlt.apply_changes_from_snapshot(...)dp.create_auto_cdc_from_snapshot_flow(...)
dlt.create_streaming_table(...)dp.create_streaming_table(...)
LIVE.<name> prefix in SQLBare name (SELECT FROM name for batch, SELECT FROM STREAM(name) for streaming). LIVE. will error in modern pipelines.
CREATE LIVE TABLE / CREATE STREAMING LIVE TABLECREATE OR REFRESH MATERIALIZED VIEW / CREATE OR REFRESH STREAMING TABLE.
CREATE TEMPORARY LIVE VIEW (a.k.a. CREATE LIVE VIEW)CREATE TEMPORARY VIEW. Exception: CREATE TEMPORARY VIEW does NOT support CONSTRAINT clauses for expectations — for the rare case where you need expectations on a temp view, CREATE LIVE VIEW is retained. See temporary-view-sql.md and expectations-sql.md.
APPLY CHANGES INTO ... FROM STREAM ... (SQL)AUTO CDC INTO ... FROM STREAM ...
partition_cols=[...] / PARTITIONED BY (...) + ZORDERcluster_by=[...] / CLUSTER BY (...) (Liquid Clustering).
input_file_name()_metadata.file_path (SQL) / F.col("_metadata.file_path") (Python).
target=... parameter on create_streaming_table / pipeline configschema=...

Performance

Language-specific guides

  • Before writing Python, first read Python basics: decorators, functions, and critical rules.
  • Before writing SQL, first read SQL basics: statements and critical rules.

Project Setup

  • Language: Keep an existing pipeline's language. For new pipelines default to SQL; use Python when the user asks for it or needs UDFs, ML inference, or Python-only features. If the request is ambiguous, ask.
  • New pipeline: Read setup-pipeline for how to scope and build it.
  • Workflow: Pick before scaffolding.
    • A: Standalone project (DAB), no existing databricks.yml → 1-project-initialization-with-dab. Default for production-bound work.
    • B: Pipeline in an existing bundle → add resources/<name>.pipeline.yml; see Workflow B in the same reference.
    • C: Rapid CLI iteration (no bundle) for prototypes and demos → 2-rapid-iteration-with-cli.
  • Structure: Medallion (Bronze → Silver → Gold) with just a few tables unless the user says otherwise. One dataset per file, named after the dataset, in src/ or transformations/. In Gold, keep the dimensions consumers filter by (e.g., every dashboard filter).
  • Settings: For create/update JSON, compute, channels, and multi-schema setups, read pipeline-settings.

Running a Pipeline

  • DAB (A/B): Code changes take effect only after databricks bundle deploy; deploy before any run, dry run, or selective refresh. Sequence: databricks bundle validate, databricks bundle deploy -t dev, databricks bundle run <pipeline_name> -t dev.
  • CLI (C): Re-upload with databricks workspace import-dir ... --overwrite after every change, then databricks pipelines start-update.
  • Poll the update ID, not the top-level pipeline state, as described in the workflow reference. A serverless update can stay INITIALIZING for a few minutes on cold start; don't cancel it.
  • Dry run (--validate-only) is a faster way to confirm the correctness of the pipeline code.
  • Selective refresh is preferred when you only need to run one table; its dependencies must already be materialized.
  • Full refresh is the most expensive and dangerous option and can lead to data loss (it resets streaming state); use it only when really necessary, and always as a follow-up the user must explicitly select.
  • Terminology: Never say "run a dry run" — the phrase is redundant. Say "do a dry run" or "start a dry run" instead.
  • To perform a rewind pipeline operation, first read rewind-pipeline.
  • To refresh or reset-and-refresh a single flow of a table (rather than the whole table), first read flow-reset-refresh. Resetting a flow cannot be undone; like a full refresh, the user must explicitly select it.

Pipeline Settings

  • To answer questions about or change pipeline settings, read pipeline-settings.md.
  • Make a pipeline continuous / always-on / "run continuously", or set its maintenance window → do NOT just set continuous: true; read continuous-pipeline.md first, as the mechanism depends on the pipeline's current mode.

Databricks Development Guide

This skill provides guidance for Databricks SDK, Databricks Connect, CLI, and REST API.

SDK Documentation: https://databricks-sdk-py.readthedocs.io/en/latest/ GitHub Repository: https://github.com/databricks/databricks-sdk-py


Environment Setup

  • Use existing virtual environment at .venv or use uv to create one
  • For Spark operations: uv pip install databricks-connect
  • For SDK operations: uv pip install databricks-sdk
  • Databricks CLI version should be 0.278.0 or higher

Configuration

  • Default profile name: DEFAULT
  • Config file: ~/.databrickscfg
  • Environment variables: DATABRICKS_HOST, DATABRICKS_TOKEN

Databricks Connect (Spark Operations)

Use databricks-connect for running Spark code locally against a Databricks cluster.

from databricks.connect import DatabricksSession

# Auto-detects 'DEFAULT' profile from ~/.databrickscfg
spark = DatabricksSession.builder.getOrCreate()

# With explicit profile
spark = DatabricksSession.builder.profile("MY_PROFILE").getOrCreate()

# Use spark as normal
df = spark.sql("SELECT * FROM catalog.schema.table")
df.show()

IMPORTANT: Do NOT set .master("local[*]") - this will cause issues with Databricks Connect.


Direct REST API Access

For operations not yet in SDK or overly complex via SDK, use direct REST API:

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

# Direct API call using authenticated client
response = w.api_client.do(
    method="GET",
    path="/api/2.0/clusters/list"
)

# POST with body
response = w.api_client.do(
    method="POST",
    path="/api/2.0/jobs/run-now",
    body={"job_id": 123}
)

When to use: Prefer SDK methods when available. Use api_client.do for:

  • New API endpoints not yet in SDK
  • Complex operations where SDK abstraction is problematic
  • Debugging/testing raw API responses

Databricks CLI

# Check version (should be >= 0.278.0)
databricks --version

# Use specific profile
databricks --profile MY_PROFILE clusters list

# Common commands
databricks clusters list
databricks jobs list
databricks workspace list /Users/me

SDK Reference

For a per-API table of method signatures and direct doc URLs (Clusters, Jobs, SQL warehouses, Unity Catalog, Serving, Vector Search, etc.), see references/doc-index.md. Worked examples for the main API surfaces live under examples/.

SDK Documentation Architecture

The SDK documentation follows a predictable URL pattern:

Base: https://databricks-sdk-py.readthedocs.io/en/latest/

Workspace APIs:  /workspace/{category}/{service}.html
Account APIs:    /account/{category}/{service}.html
Authentication:  /authentication.html
DBUtils:         /dbutils.html

Workspace API Categories

CategoryServices
computeclusters, cluster_policies, command_execution, instance_pools, libraries
catalogcatalogs, schemas, tables, volumes, functions, storage_credentials, external_locations
jobsjobs
sqlwarehouses, statement_execution, queries, alerts, dashboards
servingserving_endpoints
vectorsearchvector_search_indexes, vector_search_endpoints
pipelinespipelines
workspacerepos, secrets, workspace, git_credentials
filesfiles, dbfs
mlexperiments, model_registry

Authentication

Doc: https://databricks-sdk-py.readthedocs.io/en/latest/authentication.html

Environment Variables

DATABRICKS_HOST=https://your-workspace.cloud.databricks.com
DATABRICKS_TOKEN=dapi...  # Personal Access Token

Code Patterns

# Auto-detect credentials from environment
from databricks.sdk import WorkspaceClient
w = WorkspaceClient()

# Explicit token auth
w = WorkspaceClient(
    host="https://your-workspace.cloud.databricks.com",
    token="dapi..."
)

# Azure Service Principal
w = WorkspaceClient(
    host="https://adb-xxx.azuredatabricks.net",
    azure_workspace_resource_id="/subscriptions/.../resourceGroups/.../providers/Microsoft.Databricks/workspaces/...",
    azure_tenant_id="tenant-id",
    azure_client_id="client-id",
    azure_client_secret="secret"
)

# Use a named profile from ~/.databrickscfg
w = WorkspaceClient(profile="MY_PROFILE")

Core API Reference

Clusters API

Doc: https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html

# List all clusters
for cluster in w.clusters.list():
    print(f"{cluster.cluster_name}: {cluster.state}")

# Get cluster details
cluster = w.clusters.get(cluster_id="0123-456789-abcdef")

# Create a cluster (returns Wait object)
wait = w.clusters.create(
    cluster_name="my-cluster",
    spark_version=w.clusters.select_spark_version(latest=True),
    node_type_id=w.clusters.select_node_type(local_disk=True),
    num_workers=2
)
cluster = wait.result()  # Wait for cluster to be running

# Or use create_and_wait for blocking call
cluster = w.clusters.create_and_wait(
    cluster_name="my-cluster",
    spark_version="14.3.x-scala2.12",
    node_type_id="i3.xlarge",
    num_workers=2,
    timeout=timedelta(minutes=30)
)

# Start/stop/delete
w.clusters.start(cluster_id="...").result()
w.clusters.stop(cluster_id="...")
w.clusters.delete(cluster_id="...")

Jobs API

Doc: https://databricks-sdk-py.readthedocs.io/en/latest/workspace/jobs/jobs.html

from databricks.sdk.service.jobs import Task, NotebookTask

# List jobs
for job in w.jobs.list():
    print(f"{job.job_id}: {job.settings.name}")

# Create a job
created = w.jobs.create(
    name="my-job",
    tasks=[
        Task(
            task_key="main",
            notebook_task=NotebookTask(notebook_path="/Users/me/notebook"),
            existing_cluster_id="0123-456789-abcdef"
        )
    ]
)

# Run a job now
run = w.jobs.run_now_and_wait(job_id=created.job_id)
print(f"Run completed: {run.state.result_state}")

# Get run output
output = w.jobs.get_run_output(run_id=run.run_id)

SQL Statement Execution

Doc: https://databricks-sdk-py.readthedocs.io/en/latest/workspace/sql/statement_execution.html

# Execute SQL query
response = w.statement_execution.execute_statement(
    warehouse_id="abc123",
    statement="SELECT * FROM catalog.schema.table LIMIT 10",
    wait_timeout="30s"
)

# Check status and get results
if response.status.state == StatementState.SUCCEEDED:
    for row in response.result.data_array:
        print(row)

# For large results, fetch chunks
chunk = w.statement_execution.get_statement_result_chunk_n(
    statement_id=response.statement_id,
    chunk_index=0
)

SQL Warehouses

Doc: https://databricks-sdk-py.readthedocs.io/en/latest/workspace/sql/warehouses.html

# List warehouses
for wh in w.warehouses.list():
    print(f"{wh.name}: {wh.state}")

# Get warehouse
warehouse = w.warehouses.get(id="abc123")

# Create warehouse
created = w.warehouses.create_and_wait(
    name="my-warehouse",
    cluster_size="Small",
    max_num_clusters=1,
    auto_stop_mins=15
)

# Start/stop
w.warehouses.start(id="abc123").result()
w.warehouses.stop(id="abc123").result()

Unity Catalog - Tables

Doc: https://databricks-sdk-py.readthedocs.io/en/latest/workspace/catalog/tables.html

# List tables in a schema
for table in w.tables.list(catalog_name="main", schema_name="default"):
    print(f"{table.full_name}: {table.table_type}")

# Get table info
table = w.tables.get(full_name="main.default.my_table")
print(f"Columns: {[c.name for c in table.columns]}")

# Check if table exists
exists = w.tables.exists(full_name="main.default.my_table")

Unity Catalog - Catalogs & Schemas

Doc (Catalogs): https://databricks-sdk-py.readthedocs.io/en/latest/workspace/catalog/catalogs.html Doc (Schemas): https://databricks-sdk-py.readthedocs.io/en/latest/workspace/catalog/schemas.html

# List catalogs
for catalog in w.catalogs.list():
    print(catalog.name)

# Create catalog
w.catalogs.create(name="my_catalog", comment="Description")

# List schemas
for schema in w.schemas.list(catalog_name="main"):
    print(schema.name)

# Create schema
w.schemas.create(name="my_schema", catalog_name="main")

Volumes

Doc: https://databricks-sdk-py.readthedocs.io/en/latest/workspace/catalog/volumes.html

from databricks.sdk.service.catalog import VolumeType

# List volumes
for vol in w.volumes.list(catalog_name="main", schema_name="default"):
    print(f"{vol.full_name}: {vol.volume_type}")

# Create managed volume
w.volumes.create(
    catalog_name="main",
    schema_name="default",
    name="my_volume",
    volume_type=VolumeType.MANAGED
)

# Read volume info
vol = w.volumes.read(name="main.default.my_volume")

Files API

Doc: https://databricks-sdk-py.readthedocs.io/en/latest/workspace/files/files.html

# Upload file to volume
w.files.upload(
    file_path="/Volumes/main/default/my_volume/data.csv",
    contents=open("local_file.csv", "rb")
)

# Download file
with w.files.download(file_path="/Volumes/main/default/my_volume/data.csv") as f:
    content = f.read()

# List directory contents
for entry in w.files.list_directory_contents("/Volumes/main/default/my_volume/"):
    print(f"{entry.name}: {entry.is_directory}")

# Upload/download with progress (parallel)
w.files.upload_from(
    file_path="/Volumes/main/default/my_volume/large.parquet",
    source_path="/local/path/large.parquet",
    use_parallel=True
)

w.files.download_to(
    file_path="/Volumes/main/default/my_volume/large.parquet",
    destination="/local/output/",
    use_parallel=True
)

Serving Endpoints (Model Serving)

Doc: https://databricks-sdk-py.readthedocs.io/en/latest/workspace/serving/serving_endpoints.html

# List endpoints
for ep in w.serving_endpoints.list():
    print(f"{ep.name}: {ep.state}")

# Get endpoint
endpoint = w.serving_endpoints.get(name="my-endpoint")

# Query endpoint
response = w.serving_endpoints.query(
    name="my-endpoint",
    inputs={"prompt": "Hello, world!"}
)

# For chat/completions endpoints
response = w.serving_endpoints.query(
    name="my-chat-endpoint",
    messages=[{"role": "user", "content": "Hello!"}]
)

# Get OpenAI-compatible client
openai_client = w.serving_endpoints.get_open_ai_client()

Vector Search

Doc (Indexes): https://databricks-sdk-py.readthedocs.io/en/latest/workspace/vectorsearch/vector_search_indexes.html Doc (Endpoints): https://databricks-sdk-py.readthedocs.io/en/latest/workspace/vectorsearch/vector_search_endpoints.html

query_index uses the workspace query URL, capped at a few tens of QPS. For apps and production traffic, use the databricks-vector-search skill (AISearchClient with a service principal and the index URL).

# List vector search indexes
for idx in w.vector_search_indexes.list_indexes(endpoint_name="my-vs-endpoint"):
    print(idx.name)

# Query index
results = w.vector_search_indexes.query_index(
    index_name="main.default.my_index",
    columns=["id", "text", "embedding"],
    query_text="search query",
    num_results=10
)
for doc in results.result.data_array:
    print(doc)

Pipelines (Delta Live Tables)

Doc: https://databricks-sdk-py.readthedocs.io/en/latest/workspace/pipelines/pipelines.html

# List pipelines
for pipeline in w.pipelines.list_pipelines():
    print(f"{pipeline.name}: {pipeline.state}")

# Get pipeline
pipeline = w.pipelines.get(pipeline_id="abc123")

# Start pipeline update
w.pipelines.start_update(pipeline_id="abc123")

# Stop pipeline
w.pipelines.stop_and_wait(pipeline_id="abc123")

Secrets

Doc: https://databricks-sdk-py.readthedocs.io/en/latest/workspace/workspace/secrets.html

# List secret scopes
for scope in w.secrets.list_scopes():
    print(scope.name)

# Create scope
w.secrets.create_scope(scope="my-scope")

# Put secret
w.secrets.put_secret(scope="my-scope", key="api-key", string_value="secret123")

# Get secret (returns GetSecretResponse with value)
secret = w.secrets.get_secret(scope="my-scope", key="api-key")

# List secrets in scope (metadata only, not values)
for s in w.secrets.list_secrets(scope="my-scope"):
    print(s.key)

DBUtils

Doc: https://databricks-sdk-py.readthedocs.io/en/latest/dbutils.html

# Access dbutils through WorkspaceClient
dbutils = w.dbutils

# File system operations
files = dbutils.fs.ls("/")
dbutils.fs.cp("dbfs:/source", "dbfs:/dest")
dbutils.fs.rm("dbfs:/path", recurse=True)

# Secrets (same as w.secrets but dbutils interface)
value = dbutils.secrets.get(scope="my-scope", key="my-key")

Common Patterns

CRITICAL: Async Applications (FastAPI, etc.)

The Databricks SDK is fully synchronous. All calls block the thread. In async applications (FastAPI, asyncio), you MUST wrap SDK calls with asyncio.to_thread() to avoid blocking the event loop.

import asyncio
from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

# WRONG - blocks the event loop
async def get_clusters_bad():
    return list(w.clusters.list())  # BLOCKS!

# CORRECT - runs in thread pool
async def get_clusters_good():
    return await asyncio.to_thread(lambda: list(w.clusters.list()))

# CORRECT - for simple calls
async def get_cluster(cluster_id: str):
    return await asyncio.to_thread(w.clusters.get, cluster_id)

# CORRECT - FastAPI endpoint
from fastapi import FastAPI
app = FastAPI()

@app.get("/clusters")
async def list_clusters():
    clusters = await asyncio.to_thread(lambda: list(w.clusters.list()))
    return [{"id": c.cluster_id, "name": c.cluster_name} for c in clusters]

@app.post("/query")
async def run_query(sql: str, warehouse_id: str):
    # Wrap the blocking SDK call
    response = await asyncio.to_thread(
        w.statement_execution.execute_statement,
        statement=sql,
        warehouse_id=warehouse_id,
        wait_timeout="30s"
    )
    return response.result.data_array

Note: WorkspaceClient().config.host is NOT a network call - it just reads config. No need to wrap property access.


Wait for Long-Running Operations

from datetime import timedelta

# Pattern 1: Use *_and_wait methods
cluster = w.clusters.create_and_wait(
    cluster_name="test",
    spark_version="14.3.x-scala2.12",
    node_type_id="i3.xlarge",
    num_workers=2,
    timeout=timedelta(minutes=30)
)

# Pattern 2: Use Wait object
wait = w.clusters.create(...)
cluster = wait.result()  # Blocks until ready

# Pattern 3: Manual polling with callback
def progress(cluster):
    print(f"State: {cluster.state}")

cluster = w.clusters.wait_get_cluster_running(
    cluster_id="...",
    timeout=timedelta(minutes=30),
    callback=progress
)

Pagination

# All list methods return iterators that handle pagination automatically
for job in w.jobs.list():  # Fetches all pages
    print(job.settings.name)

# For manual control
from databricks.sdk.service.jobs import ListJobsRequest
response = w.jobs.list(limit=10)
for job in response:
    print(job)

Error Handling

from databricks.sdk.errors import NotFound, PermissionDenied, ResourceAlreadyExists

try:
    cluster = w.clusters.get(cluster_id="invalid-id")
except NotFound:
    print("Cluster not found")
except PermissionDenied:
    print("Access denied")

When Uncertain

If I'm unsure about a method, I should:

  1. Check the documentation URL pattern:

    • https://databricks-sdk-py.readthedocs.io/en/latest/workspace/{category}/{service}.html
  2. Common categories:

    • Clusters: /workspace/compute/clusters.html
    • Jobs: /workspace/jobs/jobs.html
    • Tables: /workspace/catalog/tables.html
    • Warehouses: /workspace/sql/warehouses.html
    • Serving: /workspace/serving/serving_endpoints.html
  3. Fetch and verify before providing guidance on parameters or return types.


Quick Reference Links

APIDocumentation URL
Authenticationhttps://databricks-sdk-py.readthedocs.io/en/latest/authentication.html
Clustershttps://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html
Jobshttps://databricks-sdk-py.readthedocs.io/en/latest/workspace/jobs/jobs.html
SQL Warehouseshttps://databricks-sdk-py.readthedocs.io/en/latest/workspace/sql/warehouses.html
Statement Executionhttps://databricks-sdk-py.readthedocs.io/en/latest/workspace/sql/statement_execution.html
Tableshttps://databricks-sdk-py.readthedocs.io/en/latest/workspace/catalog/tables.html
Catalogshttps://databricks-sdk-py.readthedocs.io/en/latest/workspace/catalog/catalogs.html
Schemashttps://databricks-sdk-py.readthedocs.io/en/latest/workspace/catalog/schemas.html
Volumeshttps://databricks-sdk-py.readthedocs.io/en/latest/workspace/catalog/volumes.html
Fileshttps://databricks-sdk-py.readthedocs.io/en/latest/workspace/files/files.html
Serving Endpointshttps://databricks-sdk-py.readthedocs.io/en/latest/workspace/serving/serving_endpoints.html
Vector Searchhttps://databricks-sdk-py.readthedocs.io/en/latest/workspace/vectorsearch/vector_search_indexes.html
Pipelineshttps://databricks-sdk-py.readthedocs.io/en/latest/workspace/pipelines/pipelines.html
Secretshttps://databricks-sdk-py.readthedocs.io/en/latest/workspace/workspace/secrets.html
DBUtilshttps://databricks-sdk-py.readthedocs.io/en/latest/dbutils.html

Related Skills

  • databricks-core - profile and authentication setup
  • databricks-dabs - deploying resources via DABs
  • databricks-jobs - job orchestration patterns
  • databricks-unity-catalog - catalog governance
  • databricks-model-serving - serving endpoint management
  • databricks-vector-search - vector index operations
  • databricks-lakebase - managed PostgreSQL with autoscaling + branching

Databricks Sandbox

Databricks Sandbox is a serverless, isolated compute environment for humans and agents to run untrusted code. Drive it from code with the Databricks SDK, available in Python, Java, Go, and JavaScript. Lifecycle, file operations, and synchronous execute_command_sync are available in all four SDKs; streaming execute_command is currently Python-only. The examples in this skill use the Python SDK.

SDK reference (Python): https://databricks-sdk-py.readthedocs.io/en/latest/workspace/sandbox/sandbox.html Product docs: https://docs.databricks.com/compute/serverless/sandbox SDK user guide: https://docs.databricks.com/compute/serverless/sandbox-usage-guide REST API reference: https://docs.databricks.com/api/sandbox/v1/sandbox

Other language SDKs: Java, Go, JavaScript.

For the full per-method signature table, see references/sdk-reference.md.

Environment Setup

  • Install the SDK with the sandbox extra: uv pip install "databricks-sdk[sandbox]>=0.154.0". The sandbox extra supplies the gRPC dependencies that streaming execute_command needs (the plain databricks-sdk package omits them); lifecycle, file operations, and unary execute_command_sync work without the extra. 0.154.0 is the first release with the file-operation methods used below.
  • Authentication, profiles, and ~/.databrickscfg are covered by the databricks-core skill. The sandbox client authenticates like any other WorkspaceClient.
  • Your workspace must be in a supported region (see Features with limited regional availability) and have the Databricks Sandbox preview enabled on the workspace Previews page.

Resource Naming

A sandbox's resource name is sandboxes/<sandbox_id>. create_sandbox takes the bare sandbox_id; every other call takes the full name.

Lifecycle

Lifecycle calls return before the transition finishes (create_sandbox returns while PENDING, stop_sandbox while STOPPING), so poll get_sandbox(name).status.state and wait for the target state before depending on it.

import time

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.sandbox import ComputeSpec, Sandbox, SandboxSpec, SandboxState
from google.protobuf.duration_pb2 import Duration

w = WorkspaceClient()

def wait_for_state(name: str, state: SandboxState, timeout: float = 300) -> Sandbox:
    deadline = time.monotonic() + timeout
    while time.monotonic() < deadline:
        sandbox = w.sandbox.get_sandbox(name=name)
        if sandbox.status and sandbox.status.state == state:
            return sandbox
        time.sleep(2)
    raise TimeoutError(f"{name} did not reach {state}")

# Create returns while the sandbox is still PENDING; wait for RUNNING before use.
# A sandbox bills hourly while running; set inactivity_timeout so it auto-stops when
# idle.
# Sandboxes can be updated with update_sandbox. The only accepted update paths
# are display_name and spec.compute.inactivity_timeout.
created = w.sandbox.create_sandbox(
    sandbox=Sandbox(spec=SandboxSpec(compute=ComputeSpec(inactivity_timeout=Duration(seconds=3600)))),
    sandbox_id="my-sandbox",
)
name = created.name  # "sandboxes/my-sandbox"
wait_for_state(name, SandboxState.SANDBOX_STATE_RUNNING)

# Inspect
sandbox = w.sandbox.get_sandbox(name=name)
for s in w.sandbox.list_sandboxes():
    print(s.name)

# Stop keeps the home directory; it returns while STOPPING, so wait for STOPPED
# before starting again.
w.sandbox.stop_sandbox(name=name)
wait_for_state(name, SandboxState.SANDBOX_STATE_STOPPED)
w.sandbox.start_sandbox(name=name)
wait_for_state(name, SandboxState.SANDBOX_STATE_RUNNING)

# Delete is terminal.
w.sandbox.delete_sandbox(name=name)

Stopping preserves only the home directory (/home/sandbox-agent, which holds up to 100 GB); data written elsewhere does not survive a stop (up to 10 GB of non-home storage, cleared on stop). Deleting a sandbox is terminal.

A stopped sandbox (previously by stop_sandbox()) can be restarted by explicit start_sandbox() or by running execute_command / execute_command_sync against the sandbox. For sandbox creation or restart through create_sandbox() and start_sandbox(), wait for the sandbox status to reach Running before using it; execute_command / execute_command_sync auto-starts a stopped sandbox and wait for it to reach Running, and no sandbox status polling is required.

Running Commands

Pick the mode by workload:

ModeMethodUse for
Synchronousexecute_command_syncShort commands where you only need the final stdout/stderr/exit code
Streamingexecute_commandLong-running or interactive commands where you consume output as it is produced

Streaming execute_command (the CommandProcess handle) is currently available only in the Python SDK; execute_command_sync is available in all SDKs.

cmd runs directly (not through a shell), so to use shell features invoke /bin/bash -c "..." explicitly via args.

Synchronous

from google.protobuf.duration_pb2 import Duration

resp = w.sandbox.execute_command_sync(
    "sandboxes/my-sandbox",
    "/bin/bash",
    args=["-c", "echo hello && whoami"],
    execution_timeout=Duration(seconds=30),
)
print(resp.status, resp.exit_code, resp.truncated, resp.stdout, resp.stderr)
# Check resp.status for the terminal status: exit_code is unset when the command is
# killed (e.g. timed out) or fails to start. resp.truncated is True when the output
# hit the sync size cap - use execute_command (streaming) for large output.

Streaming

execute_command returns a CommandProcess handle once the command has started. Iterate stdout / stderr for live output and call wait() for the exit code; use it as a context manager to release the client-side stream and reader thread on exit. Closing the handle (or leaving the with block) tears down the client side only - the command keeps running in the sandbox. There is no per-command cancel API; to actually stop it, read its server-side PID from proc.pid and kill it with a separate command, e.g. w.sandbox.execute_command_sync(name, "/bin/kill", args=[str(proc.pid)]).

with w.sandbox.execute_command(
    "sandboxes/my-sandbox",
    "/bin/bash",
    args=["-c", "for i in 1 2 3; do echo line $i; sleep 1; done"],
) as proc:
    for line in proc.stdout:
        print(line, end="")
    exit_code = proc.wait()  # also available as proc.exit_code once finished

stdout / stderr default to StreamMode.PIPE (buffered for reading); pass StreamMode.DEVNULL to discard, or StreamMode.STDOUT on stderr to merge it into stdout. Pass text=False for bytes instead of decoded str. Reattach to a still-running command with connect_command(name, command_id), and review history with list_commands(parent="sandboxes/my-sandbox").

Files

Manage the sandbox filesystem without a shell. All methods take parent (the sandbox name) plus a path. Write under the sandbox user's home directory, /home/sandbox-agent (the only persisted, writable tree; /home itself is root-owned, and paths outside the home directory are cleared on stop).

w.sandbox.create_directory(parent="sandboxes/my-sandbox", path="/home/sandbox-agent/data", parents=True)
for entry in w.sandbox.list_directory_contents(parent="sandboxes/my-sandbox", path="/home/sandbox-agent/data"):
    print(entry.path)
w.sandbox.get_metadata(parent="sandboxes/my-sandbox", path="/home/sandbox-agent/data")
w.sandbox.rename_path(parent="sandboxes/my-sandbox", source="/home/sandbox-agent/a", destination="/home/sandbox-agent/b")
w.sandbox.delete_path(parent="sandboxes/my-sandbox", path="/home/sandbox-agent/b", recursive=True)

Use the sandbox execute_command APIs to move data in and out; native API/SDK support for uploading and downloading file contents (local <-> sandbox) is not yet available.

When to Use

  • Agent / untrusted code execution - run model-generated code off the host, in isolation.
  • Ephemeral build/test environments - create, use, delete.

Serverless Migration API

FIRST: Use the parent databricks-core skill for CLI installation, authentication, and profile selection.

Use the Compute Migrations API through the Databricks CLI to migrate a job to serverless. The service analyzes the job, backs it up, builds a serverless working copy, and optionally runs it in the background. Your part is to track the migration, review the working copy, present the changes to the user, and carry out their decision. Applying the changes to the original job requires the user's explicit approval.

When to Use This Skill

Use it when the user asks you to:

  • migrate one job or many jobs to serverless,
  • write a script that starts and tracks migrations,
  • check on, approve, reject, or cancel migrations already in progress, or
  • find the backup a migration took of a job.

Use databricks-serverless-migration for direct code edits, notebooks, pipelines, JAR tasks, or a job this service cannot migrate. For job definitions maintained in source control, read Jobs managed from source.

See the migration guide for supported workloads and limits. A successful create starts the migration; it does not establish that the workload is compatible. Incompatibilities can be discovered during planning or working-copy creation.

Migrate One Job

  1. Check the prerequisites once per workspace.
  2. Confirm the job and whether to set auto_approve or run_validation (see Settings chosen at creation).
  3. Start the migration with create, and record its migration ID and request_id.
  4. Track progress until it reaches a review gate or a terminal state.
  5. At each review gate, review the working copy, present your findings to the user, and approve or reject as the user decides.
  6. After MIGRATION_COMPLETE, verify the original job and report the result and the backup folder.

For a list of jobs, use Bulk migration to repeat this workflow and track each job's decisions.

Prerequisites

  1. CLI v1.24.0 or newer: check databricks --version and databricks compute-migrations --help. The command group and the verbs below must be available. If a command is missing, upgrade using databricks-core.
  2. Compute Agent preview enabled in the workspace. A workspace admin enables it under Previews. Without it, every call fails with PERMISSION_DENIED ("Migration APIs are not enabled for this workspace").
  3. CAN_MANAGE on each job you migrate.

The CLI help is generated from the API and is authoritative for command names, arguments, and flags. Run databricks compute-migrations <command> --help before first use of each command. Pass request bodies with --json and read responses with -o json.

Lifecycle

Prepare working copy
  → TRANSLATION_READY (skipped when auto_approve=true)
  → Optional validation run (run_validation=true)
  → MIGRATION_REVIEW (execution approval required)
  → Apply changes → MIGRATION_COMPLETE
  1. Planning. The service reads the job, its notebooks and files, and its cluster configuration, and works out the serverless changes: compute, dependencies, Spark configs, and code changes. The working copy is the first thing you review.
  2. Backup and working copy. The service saves a backup of the job and its workspace files (see Backups), then creates a working copy: a new job named [Serverless Migration - <timestamp>] <job name>, tagged serverless_migration_source_job_id=<original job id>, running copies of the job's notebooks and files. The original job is not modified until you approve execution.
  3. Gate 1, TRANSLATION_READY. Skipped when auto_approve=true. Otherwise, review the working copy. Approve to continue, or reject with a reason to request changes.
  4. Validation run (only with run_validation). Databricks runs the working-copy job once. The outcome is recorded in validation_report. A failed run does not stop the migration: it moves to MIGRATION_REVIEW with passed: false.
  5. Gate 2, MIGRATION_REVIEW. Final approval to update the original job. Approving execution starts applying the working copy's settings and file contents to the original job, keeping its job ID, schedule, and permissions.
  6. Complete. Verify the original job (see After completion).

Use either rejection command to request changes, with a specific rejection_reason. The same migration returns to planning and revises its working copy. Read migration_report again and review the current copy before approving it. Revisions are limited, and repeated rejections eventually fail the migration. When the user wants to stop the migration, use cancel-migration (see Cancellation).

Settings Chosen at Creation

auto_approve and run_validation are fixed when the migration is created and cannot be changed. Both default to false. Set either one only when the user asks for it.

FieldEffectRisk
auto_approveSkips gate 1 (TRANSLATION_READY). Gate 2 still waits for a decision.With run_validation, the working copy runs before the user reviews it.
run_validationRuns the working-copy job once before gate 2.The run is real: see below.

A validation run is not sandboxed. The working copy runs in the same workspace against the same tables, volumes, checkpoints, and external systems as the original job, with the agent's changes applied. If the job is not idempotent, or has side effects, the validation run produces them. Examples: overwriting or appending to production tables, merges, sending emails or notifications, calling external APIs, triggering downstream jobs, or advancing streaming offsets and checkpoints. Enable run_validation only after the user confirms that one extra run of this job, right now, is safe, for example because the job is idempotent or writes only to test outputs. Never turn it on across a batch of jobs without that confirmation for each job.

Commands

OperationCommandNotes
Startcreate --json '<migration>' --request-id <uuid>asset_type is JOB; asset_uri is jobs/<job_id>.
Read oneget <migration_id>Current state, reports, etag.
Read manylistReturns a JSON array with -o json and paginates automatically. Filter the returned state values locally. Eventually consistent: confirm with get before acting.
Gate 1approve-migration-translation <migration_id> / reject-migration-translation <migration_id> <rejection_reason>Reject takes the reason as a positional argument or as rejection_reason in --json. Approve accepts optional --additional-feedback.
Gate 2approve-migration-execution <migration_id> / reject-migration-execution <migration_id> <rejection_reason>Same arguments and flags as gate 1.
Stopcancel-migration <migration_id>Any non-terminal state. Optional cancellation_reason. Does not undo applied changes; see Cancellation.
Remove recorddelete <migration_id>Hides the record and frees the job for a new migration. Rolls nothing back and cleans nothing up. Use cancel-migration to stop an in-flight migration.

All commands are under databricks compute-migrations. Start a migration:

databricks compute-migrations create -o json --request-id "<uuid>" --json '{
  "asset_type": "JOB",
  "asset_uri": "jobs/<job_id>",
  "auto_approve": false,
  "run_validation": false
}'

Generate a fresh request_id per job and reuse it when retrying that job's create after a timeout or network error. The server returns the existing migration instead of creating a duplicate.

Pass the etag from your latest get on every approve, reject, and cancel, so the call fails rather than acting on a state you have not seen:

databricks compute-migrations reject-migration-translation <migration_id> -o json --json '{
  "etag": "<etag>",
  "rejection_reason": "Keep the existing output table names; task ingest writes to a different table in the working copy."
}'
ErrorMeaningAction
ALREADY_EXISTSThe job has a migration in progress, or one that already migrated it.Find and confirm the existing migration as described below.
INVALID_PARAMETER_VALUEBad request, or the job cannot be migrated.Read the message; do not retry unchanged.
PERMISSION_DENIEDNo CAN_MANAGE on the job, or the preview is not enabled.Report to the user.
FAILED_PRECONDITIONGate call made in the wrong state.get, then act on the current state.
ABORTEDThe etag is stale.get, re-check the state, retry with the new etag.

After ALREADY_EXISTS, take the migration ID from the error and confirm its asset_uri and state with get. If the existing migration is MIGRATION_COMPLETE, report the job as already migrated. Do not delete the record and create another migration unless the user explicitly requests it.

Tracking Progress

Migrations take minutes to hours: planning and building the working copy usually take several minutes, and a validation run takes as long as the job. Use your harness's monitoring capabilities to track progress. get returns the current migration state; for many migrations, use list and confirm each one with get before acting.

StateWhat to do
PLANNING_PENDING, PLANNING_RUNNING, REVISION_PENDINGPlanning is in progress.
TRANSLATION_PENDING, TRANSLATION_CREATINGWorking-copy creation is in progress.
VALIDATION_SCHEDULING, VALIDATION_RUNNINGValidation is in progress.
MIGRATION_PENDING, MIGRATION_RUNNING, MIGRATION_OBSERVATIONMigration is in progress.
TRANSLATION_READYReview the working copy, present it to the user, then approve or reject gate 1.
MIGRATION_REVIEWReview the working copy and validation_report, present them, then approve or reject gate 2.
MIGRATION_COMPLETEVerify the original job.
NOT_SUPPORTEDThe job cannot be migrated automatically. Report the reason from state_history, and offer databricks-serverless-migration for a manual migration.
FAILEDReport the reason from state_history. If the migration failed during MIGRATION_RUNNING, check whether the original job was partly changed (see Backups).
CANCELLEDReport the cancellation and check for applied changes or active validation runs as described in Cancellation.
Any other stateRead the migration again with get. If the state is still unrecognized, report it and stop automatic progress handling; do not assume it is in progress.

state_history lists every state entered, with transition_time, the reviewer's user_comment, and the system's internal_comment (for example, the reason for a failure).

Cancellation

Cancellation does not undo changes already applied to the original job or files, or side effects of validation runs. An operation already in progress may finish after the migration reports CANCELLED; that state alone does not confirm that all work has stopped.

If execution was approved, or state_history includes MIGRATION_PENDING or MIGRATION_RUNNING, inspect the original job and files against the saved review record and backup. Report which changes are present and anything you cannot verify. Check active working-copy runs separately with databricks jobs list-runs --job-id <working_copy_job_id> --active-only -o json. Retain the recovery artifacts if the result is unverified or work may still be active.

Reviewing a Working Copy

The API returns links to the working copy, not a diff, so build the comparison yourself:

  1. get the migration. migration_report.primary_asset.id is the working-copy job's ID.
  2. Compare job settings: databricks jobs get <original_job_id> -o json against databricks jobs get <working_copy_job_id> -o json. Expect classic compute (job_clusters, new_cluster, existing_cluster_id) to be replaced by serverless settings (environments and environment_key), libraries to move into environment dependencies, unsupported Spark configs and init scripts to be removed, and possibly a performance mode to be set. The job name prefix, the serverless_migration_source_job_id tag, and task paths pointing at the copied notebooks are expected differences.
  3. Compare code. The working-copy job's task settings name the notebooks and files it runs. Match its tasks to the original job's by task_key, and take each task's path (notebook_task.notebook_path or spark_python_task.python_file) from both jobs. For for_each_task, inspect its inner task too. Export each to a separate local file with databricks workspace export <path> --format SOURCE --file <local_file> and diff them. Also inspect utility notebooks invoked by the task notebooks for changes, such as through %run or dbutils.notebook.run.
  4. If a validation run happened, report validation_report.passed, run_status, and run_url. For a failed run, inspect it with databricks jobs get-run <run_id> -o json. Take the run ID from run_url.
  5. Summarize the changes per task, including which tasks will migrate and which will stay on classic. Flag anything beyond a compatibility fix: changed business logic, different output tables or paths, removed steps, or dropped dependencies.
  6. Before approving execution, save the reviewed job settings and exported code, including utility notebooks, in a local folder for this migration ID. Include the original task paths, the working-copy-to-original path mapping, and the tasks approved for migration or intentionally kept on classic. This review record is needed after the working copy is deleted. Refresh it after any revision and review.
  7. Get the user's decision. A rejection reason should name the task and the specific change to make.

Rules while a migration is in progress:

  • Approve gate 2 only with explicit user approval covering that job ID. A single decision may approve an explicitly identified set of jobs; the approval must cover the reviewed changes for each job.
  • Do not edit the original job. Approving execution replaces its settings and files with the working copy's, discarding later edits.
  • Do not edit the working copy to change the outcome. Reject with a reason instead.

After Completion

When the API reports MIGRATION_COMPLETE, verify the original job before reporting success:

  • Read databricks jobs get <job_id> -o json and compare it with the saved review record. Tasks approved for migration must have the reviewed serverless settings; tasks intentionally left on classic must retain their reviewed configuration. Account for the original job name and task paths, and removal of the working-copy tag.
  • Export the original task notebooks and files, including those used by nested tasks, along with the reviewed utility notebooks. Compare them with the saved reviewed code.

If approved changes are missing or differ from the review record, report the mismatch and offer to restore from the backup. Before retrying, restore any job settings or files changed by this migration from the pre-migration backup, with the user's approval. Then delete the migration (which frees the job) and create a new one. If the review record is unavailable, explain which parts of the result you could verify and which remain unverified.

The working-copy job or its copied files may remain after a successful migration. Report any leftovers and offer cleanup.

Report the migration ID, final state, original job, what changed, tasks left on classic, and the backup folder.

Backups

Before building the working copy, the service writes a backup to the workspace home of the identity that created the migration.

Files that tasks load from a Git repository are not copied; they stay in the repository. A new timestamped backup folder is written each time the service builds a working copy. Migrations never delete backups. Give the user the backup folder for every migration that reached TRANSLATION_READY or later.

There is no rollback command. For the backup location and file layout, restore instructions, and working-copy cleanup, follow Backups and rollback.

Bulk Migration

For multiple jobs, follow Bulk migration. It applies the single-job workflow to each migration, with separate tracking, settings, review records, and approval decisions.

Jobs Managed from Source

A migration changes the job in the workspace. If the job's definition comes from source, the next deploy reverts the migration. This covers jobs deployed from a Declarative Automation Bundle (settings.deployment.kind is BUNDLE) and jobs managed by Terraform or CI. For these jobs, tell the user before creating a migration. Either migrate the source (databricks-serverless-migration, databricks-dabs), or use the reviewed working copy as the reference for changes to make in the source.

Documentation

Serverless Compute Migration

FIRST: Use the parent databricks-core skill for CLI basics, authentication, and profile selection.

Analyze existing Databricks code for serverless compute compatibility and guide migration from classic clusters. The skill follows a 4-step migration lifecycle: Ingest the workload → Analyze for compatibility → Test via A/B comparison → Validate and iterate.

Prefer the Compute Migrations API for Jobs

To migrate a job, first check whether databricks-serverless-migration-api is installed and its prerequisites are available. If so, use it to start and manage the migration. The service backs up the job, builds and optionally runs a serverless working copy, and changes the original only after review. Use the manual workflow below when:

  • the API skill, databricks compute-migrations CLI commands, or Compute Agent preview are not available,
  • the workload is a standalone notebook or pipeline,
  • the service reports the job as not supported, or
  • the user wants you to make the changes yourself.

When to Use This Skill

  • Migrating notebooks, jobs, or pipelines from classic compute to serverless
  • Checking if existing code is serverless-compatible
  • Writing new code that targets serverless compute
  • Troubleshooting serverless-specific errors after migration
  • Choosing between Performance-Optimized and Standard mode

Where to Run This Skill

Runs in any agentskills.io client: on a laptop (Claude Code, Cursor, ...; the default, install via databricks aitools install), or inside a Databricks workspace via Genie Code Agent mode (per-user or workspace-wide .assistant/skills/ folder). See Install in Databricks Genie Code for the three install methods and the serverless-compute caveat (the Databricks CLI isn't pre-installed on serverless, so some deploy steps need adjustment).

If you finish a migration analysis for a user who's currently running you from a laptop client, mention the Genie Code option once at the end — many users prefer iterating on migrations inside the workspace where the workload lives.

Understanding Migration Blockers

Migration blockers fall into three categories. Focus your effort on category 2 — that's where this skill helps most.

CategoryDescriptionAction
1. Feature expandingDatabricks is actively expanding support (e.g., SparkML, custom JDBC)Use the workaround now and revisit later
2. Code/config change neededYour code uses patterns that need updating for serverless (e.g., RDDs, DBFS, streaming triggers)This skill helps here — it detects these patterns and provides fixes
3. Classic-onlyWorkload requires capabilities not available on serverless (e.g., root OS access, R language)Keep on classic compute

Decision Tree: Is My Workload Ready?

Workload → Check language
├── R code → Category 3: keep on classic
├── Scala notebook cells → Category 2: port to PySpark/SQL, or compile as a JAR job task
├── Compiled Scala JAR (spark_jar_task, build.sbt/pom.xml/build.gradle) → see references/jar-migration.md
│     ├── Scala 2.12 build?             → recompile against 2.13.16
│     ├── Bundles spark-core/spark-sql? → use databricks-connect % Provided
│     └── Bundles libs on the kernel classpath? → mark % Provided (see the classpath table in references/jar-migration.md)
├── Python / SQL → Continue
    ├── Uses RDD APIs? → Category 2: rewrite to DataFrame API (see fixes below)
    ├── Uses DBFS paths? → Category 2: migrate to UC Volumes
    ├── Uses Hive Metastore? → Category 2: migrate to Unity Catalog (or use HMS Federation)
    ├── Uses df.cache/persist? → Category 1: remove and materialize to Delta (native support coming soon)
    ├── Uses streaming?
    │   ├── ProcessingTime trigger → Category 2: use AvailableNow or migrate to SDP
    │   ├── Continuous trigger → Category 2: use SDP continuous mode
    │   ├── No trigger specified → Category 2: add explicit .trigger(availableNow=True)
    │   └── AvailableNow / Once → Ready ✓
    ├── Uses init scripts? → Category 2: use Environments
    ├── Uses VPC peering? → Category 2: use NCCs / Private Link
    ├── Uses unsupported Spark configs? → Category 2: remove (serverless auto-tunes)
    ├── Uses custom JDBC drivers? → Category 2: use Lakehouse Federation or built-in JDBC
    ├── Uses Docker containers? → Category 3: use Environments for libs, or keep on classic
    └── All clear → Ready for serverless ✓

Migration Workflow

Step 1: Ingest — Gather Workload Context

Confirm the migration target is serverless compute. This skill is purpose-built for classic → serverless migrations. The checks, fixes, and workflow all target the serverless compute architecture (Spark Connect, Environments, NCCs). If the user wants to upgrade between classic DBR versions instead, this skill does not apply — classic DBR upgrades have a different compatibility surface and should follow the standard DBR upgrade guide.

Collect the full picture of what needs to migrate to serverless:

  • Read the user's notebook/script files
  • Identify the classic cluster configuration (instance type, DBR version, Spark configs, init scripts, libraries)
  • Note the networking setup (VPC peering, instance profiles, mounts)
  • Understand the workload type: batch job, streaming, interactive notebook, pipeline
  • Determine the target: the output is always a serverless compute configuration, not a classic cluster with a newer DBR

Step 1.5: Handle Multi-Notebook Workloads

If the workload spans 3+ user notebooks (exclude _resources/ setup notebooks) or any notebook exceeds ~5KB of source, process notebooks one at a time to avoid context overflow (autocompact thrashing): enumerate first (including upstream sources declared in bundle_config.py, H4), then per notebook read → run the Step 2 checklist → record a structured summary and drop the source, then synthesize the plan from the summaries. Trigger the Failure Reporting Protocol once per workload, not per notebook. Don't merge all notebooks into one file or act on all of them in one mega-turn. Full procedure (summary fields, scratch-dir layout, report aggregation): Multi-Notebook Workloads.

Step 2: Analyze — Scan for Serverless Readiness

Read notebooks before running them — do not rely on failed job runs to discover issues. A pre-run scan surfaces incompatibilities faster than iterating on error traces, and many serverless failures (hardcoded catalog references, init scripts, missing dependencies) are easy to spot statically but expensive to debug after a failed run.

Before creating or running any test job:

  1. Read every notebook and source file referenced by the job
  2. Scan for all hardcoded catalog/schema references (e.g., spark.table("main.schema.table"), spark.sql("... FROM main..."), catalog = "main")
  3. Check for dependency patterns: init scripts, local wheel files, custom install functions, %pip install lines
  4. Locate any requirements.txt or equivalent and resolve the full dependency set
  5. Flag OS-level installs (apt install, yum install) for conversion or escalation

Scan the code for patterns that are incompatible with the serverless compute architecture. These checks are serverless-specific — most of these patterns work fine on classic compute regardless of DBR version. For each issue found, report:

  • Category: Which of the 3 blocker categories it falls into
  • Severity: Blocker (must fix for serverless) / Warning (should fix) / Info (awareness)
  • Pattern: What was detected and where
  • Fix: Specific remediation targeting serverless compute

Cell-magic boundary lint (A1), high impact: in any cell with a # MAGIC %<word> line, every non-blank line must start with # MAGIC . A plain-Python comment above # MAGIC %run in the same cell makes %run fall back to IPython (File './…' not found); put migration notes in their own # MAGIC %md cell instead.

Pattern catalog: load Analyze Checklist for every Step 2 scan. It lists each check with severity and fix: A APIs (RDD, caching, MLflow/AutoML A2–A5, M1–M4), B data access (B1–B2), C streaming, D config (D1), E libraries (E1–E2), F networking, G sizing, H job-level config (H1–H4).

Required Output: Serverless Environment Specification

The migration output MUST include a Serverless Environment specification alongside migrated code. Generate this by:

  1. Scanning all import statements and %pip install lines to detect required packages
  2. Extracting init script pip install commands from the job configuration
  3. Producing a JSON block suitable for the Jobs API environments field:
{
  "environment_key": "Default",
  "spec": {
    "client": "2",
    "dependencies": ["mlflow==2.12.1", "scikit-learn==1.3.0", "xgboost==2.0.3"]
  }
}

Important: ML runtime is NOT available on serverless and ML libraries (mlflow, scikit-learn, hyperopt, xgboost, tensorflow, torch, etc.) are NOT pre-installed, so list them explicitly in the environment spec dependencies.

Step 3: Test — Two-Branch Strategy

Use separate branches for testing and production to keep test-only workarounds out of the code that ships. The test branch is a safe sandbox for experimentation; the production branch contains only changes that production actually needs.

Branch-by-branch comparison (names, base branch, PR, catalog, what each may contain): Testing Strategy.

Test branch (serverless-test-{job_name}-{timestamp}): Temporary, no PR needed.

  1. Create a branch from your current working branch
  2. Set up test data: create sampled copies of upstream tables in a test catalog using job lineage (see Testing Strategy § Test Data Setup)
  3. Parameterize the catalog so the notebook works with both test and production data (see Quick Fixes § Parameterize catalogs)
  4. Apply all compatibility fixes discovered in Step 2
  5. Create a serverless test job and run it
  6. If it fails, get the error output, debug, fix, and retry
  7. Document which changes are test workarounds vs. real compatibility fixes

Production branch (serverless-prod-{job_name}): PR required, created from master.

  1. Create a new branch from master (NOT from the test branch)
  2. Apply ONLY the real compatibility fixes — no test workarounds
  3. Apply job config changes (see Quick Fixes and Debugging)
  4. Commit and create a PR

Decision Tree: Should This Change Go to Production?

Change typeProduction?Reason
Remove incompatible Spark configsYesServerless compatibility fix
Update library versionsYesServerless compatibility fix
Replace DBFS paths with UC VolumesYesServerless compatibility fix
Remove init scripts, add EnvironmentsYesServerless compatibility fix
Fix hardcoded cluster settingsYesServerless compatibility fix
Catalog override to test catalogNoTest workaround only
Empty DataFrame handling for missing test dataNoTest workaround only
Date range limiting for faster testsNoTest workaround only

Simple test: Would production fail without this change on serverless? If yes → include. If no → test branch only.

A/B Comparison

After both branches are ready, compare classic vs serverless outputs (row count, schema, exceptAll row diff; snippet and the temporary bridge-config rule in Testing Strategy § A/B Comparison). Test against sampled copies of upstream tables in a test catalog, never production data (Test Data Setup).

Step 4: Validate — Confirm and Monitor

Once the A/B comparison passes:

  1. Merge the production branch PR
  2. Switch the production job to serverless compute
  3. Monitor cost via system tables (system.billing.usage) and budget policies
  4. Remove any temporary bridge configurations
  5. Set up budget alerts for cost visibility

Migration Deliverables

Surface all six at the end of a successful run: test branch name/URL, production branch name/URL, test job ID and run URL, classic vs serverless comparison, serverless environment spec, and change summary (production vs test-only, with reasons). What each is and why it matters: Testing Strategy § Migration Deliverables. If any deliverable is missing, the migration is incomplete — do not mark it as done.

Stopping Conditions

Do not attempt workarounds for these — surface them to the user and stop:

  • Permission failures on source tables, the test catalog, or the workspace
  • Category 3 blockers (R code, custom Spark data source JARs, features that require classic compute)
  • SQL warehouse or test catalog not available
  • Repeated failures (typically 5+) with no new information in the error trace — generate a failure report instead (see Failure Reporting Protocol)

Quick Fixes and Debugging

  • Quick Fixes: before/after snippets for DBFS → UC Volumes, RDD → DataFrame, streaming triggers (+ OOM caps for Delta, Kafka, Auto Loader), caching removal, runtime serverless detection (IS_SERVERLESS), and catalog parameterization for tests.
  • Debugging Errors: error message → fix table for failed serverless runs. Always get the actual error with w.jobs.get_run_output(run_id=...) before guessing.

Transform job config from classic to serverless

Remove job_clusters/new_cluster, add environments with serverless spec, replace job_cluster_key with environment_key, remove init_scripts. Full rules (init scripts → dependencies, Maven/JAR libraries, spark_conf triage) and before/after JSON: Configuration Guide.

Environment version mapping (match to the DBR version the workload was on):

Classic DBRServerless spec.clientPython
13.x, 14.x"1"3.10
15.x"2"3.11
16.x+"3"3.12

Performance Mode Selection

Performance-Optimized (default in UI and API; <50 s startup; higher cost; notebooks, jobs, SDP) for interactive, dev, and time-sensitive work. Standard (4-6 min startup; significantly lower cost; jobs and SDP only, must be selected explicitly) for batch ETL and scheduled pipelines. Standard mode is NOT available for notebooks. Selection guide and how to set it: Configuration Guide § Performance Mode Selection.

Serverless Defaults to Know

SettingValue
REPL VM memory8GB default (high-memory option available)
Max executors32 (Premium), 64 (Enterprise) — raise via support
Supported Spark configs6 only: spark.sql.shuffle.partitions, spark.sql.session.timeZone, spark.sql.ansi.enabled, spark.sql.files.maxPartitionBytes, spark.sql.legacy.timeParserPolicy, spark.databricks.execution.timeout
DebuggingQuery Profile (no Spark UI)
ANSI SQLEnabled by default (configurable)

Failure Reporting Protocol

When migration fails or hits an unmigratable pattern, generate a structured failure report and offer the user a one-step path to file it as a GitHub issue. This feedback loop is how the skill learns — without it, gaps go undiscovered.

The report schema, MUST-NOT-contain rules, anonymization regexes, deterministic post-serialization scrub, output template, and URL recipe are in Failure Reporting; load it before writing a report.

Decision tree: when to offer to file an issue

Offer to file a GitHub issue any time the workload cannot be fully migrated to serverless as-is. Reports from "known" patterns (R, Scala, custom JAR data sources, JVM access, third-party connectors) are just as valuable as reports from unknown patterns — they tell maintainers which gaps users hit most often, which drives prioritization.

Concretely, ALWAYS offer if any of these is true:

  1. Workload contains any Category 3 (classic-only) blocker — R, Scala notebook cells, custom JAR data sources, JVM/Py4J access, third-party connectors without serverless equivalents, native binary dependencies, etc. The fact that the pattern is "documented as Cat 3" is not a reason to skip the offer.
  2. Retries exhausted — retry_count >= max_retries (typically 5) and final status is FAILED
  3. Unknown pattern — a classic-compute construct was detected that isn't in the skill's catalog
  4. Fix didn't resolve — a known fix was applied but the workload still fails on serverless
  5. Explicit request — the user invokes /migration-report

Do NOT offer to file only when:

  • The migration succeeded fully (even after retries), or
  • The workload is already serverless-compatible and required no changes.

Rules that always apply: write the report locally to ~/.databricks-migration-skill/reports/failure-<ISO-timestamp>.json (pattern IDs and notebook_characteristics only, no code, paths, error text, identifiers, or credentials). Default output is local-only: show the path and ask the user to review. Produce the pre-filled GitHub URL or gh issue create command only in a later turn, after the user explicitly confirms and the deterministic scrub of the final JSON is clean. Never transmit the report automatically; if the user declines, log the local path and move on.

Success report

When migration succeeds, generate a per-workload report with the same structure as the failure report (minus the redaction-to-publish step), listing the patterns detected and the fixes applied for each workload. Write to ~/.databricks-migration-skill/reports/success-<timestamp>.json and tell the user the file path.

Multiple workload migration

When several independent workloads are in scope, migrate them in parallel (e.g. one subagent per workload). Keep the per-notebook steps within a single workload sequential, as in Step 1.5.

Reference Guides

Documentation

Deploying Databricks jobs and pipelines on serverless

FIRST: Use the parent databricks-core skill for CLI basics, authentication and profile selection.

This skill decides compute. Where another skill or example (for example databricks-jobs) shows cluster configuration, follow this skill instead.

Serverless is the default compute for every job, pipeline and bundle you create. If the user explicitly asks for a cluster, do what they ask and mention the serverless option in one line. When editing an existing job, keep its compute unless the user asks to change it; tasks you add follow this skill.

1. Serverless or classic

Use classic compute only for:

  • R code, or Scala in notebook cells (%scala): serverless notebooks run Python and SQL only;
  • custom Docker images;
  • OS packages (apt-get) or native libraries with no pip equivalent;
  • custom Spark data source JARs;
  • a stated hard requirement for an instance type or Databricks Runtime version;
  • an explicit user request;
  • existing code over the budget in section 5;
  • a workspace without serverless: if creating or running the job fails because serverless is not available there, put the whole job on classic and say so.

Then give only the blocked task a cluster (one small job_clusters entry with the latest LTS runtime and autoscale: {min_workers: 1, max_workers: 4}), keep every other task serverless, and state the blocker in one line. For a blocked pipeline, give the pipeline a clusters block instead of serverless: true.

Not reasons for classic: Scala or Java in a JAR task (JAR tasks run on serverless), GPUs (serverless GPU), large data, long runtimes, Kafka, cost (serverless STANDARD is on par with or cheaper than on-demand classic in most cases), or habit.

2. Put every task on serverless

  • New jobs: leave out new_cluster, job_clusters, job_cluster_key, existing_cluster_id, instance_pool_id, node_type_id, num_workers, autoscale, spark_version. A task without a cluster runs on serverless. In an existing job, keep the cluster settings it has.
  • Notebook tasks need nothing else. Python script, wheel and JAR tasks take their libraries from a job-level environment. Use the latest environment version the workspace offers, as a quoted string:
    environments:
      - environment_key: default
        spec:
          environment_version: "4"                # example; use the latest
          dependencies: ["requests==2.32.3"]   # pin versions; JARs go in java_dependencies
    tasks:
      - task_key: main
        spark_python_task: {python_file: /Workspace/.../main.py}
        environment_key: default
    
  • SQL tasks: a serverless SQL warehouse. Pipelines: serverless: true, no clusters block.
  • Streaming: .trigger(availableNow=True), the trigger serverless supports (no processingTime or continuous triggers). For always-on processing, run that stream in a continuous job (continuous: {pause_status: UNPAUSED}), which starts the next run as soon as one finishes, or use a continuous pipeline.
  • Tag every job you create for resource tracking, as other Databricks skills do: job-level tags: {"aidevkit_project": "ai-dev-kit"}. Keep tags the user already has.

3. Choose the performance mode

Set the job-level performance_target:

  • STANDARD when nothing is time-critical and cost matters more than speed: scheduled batch and ETL, nightly or weekly runs, or any job where the user states no latency need. Runs start within minutes instead of seconds and cost less. This is the default choice.
  • PERFORMANCE_OPTIMIZED only when the user names an SLA or a deadline that is tight relative to the runtime, a person waits on the result, or the job runs every 30 minutes or more often.

performance_target applies to the whole job, notebook tasks included (only interactive notebooks always run performance-optimized). Say which mode you chose and why in one line, e.g. "STANDARD: nightly batch, no deadline." Serverless GPU tasks ignore performance_target.

Pipelines have no performance_target of their own. To schedule a pipeline, trigger it from a job with a pipeline_task and set the mode on that job; the job's mode applies to the pipeline update. A continuous pipeline runs in STANDARD only when a continuous job runs it.

4. Code you write runs on serverless from the start

Serverless reads Unity Catalog tables, /Volumes/... paths, cloud paths behind a Unity Catalog external location, and the sample data under dbfs:/databricks-datasets/. It cannot read DBFS mounts (dbfs:/mnt/...) or other DBFS root paths; only those count as a DBFS blocker.

DataFrame and SQL APIs only (no RDDs, sc.*, spark.sparkContext); Unity Catalog three-part table names; /Volumes/... paths instead of DBFS mounts (dbfs:/mnt/...); availableNow streaming triggers; no .cache() or .persist(); no spark.conf.set for executor, driver or memory settings; pinned dependencies in the environment, never init scripts.

5. Existing code: a quick check, not a migration

The user is waiting for the deployment. Spend one short pass, then deploy.

  1. Scan only the files the job runs (a text search such as grep or rg; do not read the whole repo) for: sc., sparkContext, .rdd, parallelize, mapPartitions, reduceByKey, groupByKey, dbfs:/, /dbfs/, dbutils.fs.mount, .cache(, .persist(, spark.conf.set, processingTime, continuous=, writeStream, hive_metastore., GLOBAL TEMP, hivevar, init_scripts, apt-get, docker, and other languages: %scala or %r as the whole magic (# MAGIC %r, not %run), .r files and R notebooks, SparkR and sparklyr. R and Scala cells go to classic (section 1).

  2. Minor blockers: fix them and deploy on serverless if every fix is mechanical and local, keeps the logic, needs no new infrastructure, and there are about 5 small edits or fewer in total:

    • delete .cache(), .persist(), .unpersist(), setLogLevel, executor/driver/AQE spark.conf.set lines
    • move pip installs or requirements.txt into the environment, pinned
    • add .trigger(availableNow=True) to a stream that sets no trigger, in a job that runs on a schedule anyway (a stream with a processingTime or continuous trigger is a step 4 blocker)
    • replace a dbfs:/mnt/... path with a /Volumes/... path only when that volume's storage location is the mount's source, so the job reads the same data; a volume with a similar name is not enough. Otherwise the mount is a step 4 blocker.
    • CREATE GLOBAL TEMP VIEW becomes CREATE TEMP VIEW, and its global_temp.<view> reads become <view>, when the same file creates and reads the view (serverless has no global temp views); a global temp view that another task or notebook reads is a step 4 blocker
    • add the catalog to a two-part table name only when you know which Unity Catalog catalog holds the table (for example the job's default catalog); a table in hive_metastore is a step 4 blocker

    List every edit you made in your reply.

  3. RDD or SparkContext code: translate it to DataFrames and deploy on serverless only if all of these hold:

    • it is one file of about 200 lines or fewer;
    • the change is a pure API translation: the same logic, expressed with DataFrame or SQL;
    • every input path, table, output table and parameter stays exactly as it was. Never swap in another dataset, even a sample one;
    • every input is already readable on serverless (section 4: a Unity Catalog table, an existing /Volumes/... path, a cloud path behind an external location, or dbfs:/databricks-datasets/), so nothing new has to be set up.

    Save the original next to it as <name>.classic.py and point the job at the translated file. Tell the user that the translation has not been run yet and suggest one test run that compares the output with the classic version.

  4. Anything bigger: do not rewrite. Put that task on classic now (section 1), keep the other tasks serverless, and end with: "Kept on classic because . To move it to serverless, use the databricks-serverless-migration skill." Bigger means: RDD code that fails any condition in step 3 (for example it reads a DBFS mount), mounts with no volume on the same storage, Hive Metastore to Unity Catalog, recompiling a JAR, networking changes, changing what a stream does (an always-on processingTime or continuous stream), or more than about 5 edits besides a step 3 translation.

Do not stop to ask whether to migrate. Deploy, then report what you did.

Common Issues

IssueSolution
Libraries field is not supported for serverless taskMove libraries into environments[].spec.dependencies (JARs: java_dependencies) and set environment_key on the task
Serverless job starts too slowlyperformance_target: PERFORMANCE_OPTIMIZED, only if the user needs fast starts (section 3)
Only remote Spark sessions ... are supported or another RDD errorRDD/SparkContext code: section 5, step 3 or 4
[PATH_NOT_FOUND] or no access on dbfs:/mnt/...DBFS mounts do not work on serverless: use a volume on the same storage (section 5, step 2), otherwise step 4
Serverless is not enabled or not available in the workspacePut the job on classic and tell the user (section 1)
A sibling skill's example uses job_clusters for a new jobLeave the cluster block out; this skill decides compute. Existing jobs keep their compute

Related Skills

  • databricks-core - CLI basics, authentication, profiles
  • databricks-jobs - job structure, task types, triggers, notifications
  • databricks-pipelines - Lakeflow Spark Declarative Pipelines
  • databricks-serverless-migration - migrating existing classic workloads to serverless

Documentation

Databricks Setup-Local

REQUIRED FIRST: Use databricks-core for CLI, authentication, and profile selection. Never use a default profile. For an existing environment, use databricks-execution-compute.

Workflow

1. Check CLI and authentication

The CLI must be >= v1.16.0 -- the release that added --no-dbconnect and the E_PROVISION_CONFLICT failure this skill routes. Compare deterministically -- never eyeball the version (1.9.0 is older than 1.16.0, and a lexical string compare gets this wrong). Run this gate first and do not proceed if it exits non-zero:

# Subshell so a failed gate reports non-zero without closing a persistent shell.
(
  # Read stdout only -- upgrade nags on stderr carry their own semver and would win the parse.
  if ! raw="$(databricks version 2>/dev/null)"; then
    echo "STOP: databricks CLI is missing or failed to run -- install or upgrade it via databricks-core first"
    exit 1
  fi
  have="$(printf '%s\n' "$raw" | grep -oE 'v[0-9]+\.[0-9]+\.[0-9]+' | head -n1 | tr -d v)"
  [ -n "$have" ] || have="$(printf '%s\n' "$raw" | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -n1)"
  if [ -z "$have" ]; then
    echo "STOP: could not read a version from 'databricks version' -- upgrade via databricks-core first"
    exit 1
  fi
  case "$raw" in
    *-dev*)
      echo "NOTE: dev build $have -- the floor cannot be checked; ask the user to confirm this build has setup-local"
      ;;
    *)
      if [ "$(printf '%s\n%s\n' "1.16.0" "$have" | sort -V | head -n1)" != "1.16.0" ]; then
        echo "STOP: databricks CLI $have is older than v1.16.0, the floor this skill requires -- older CLIs can run setup-local but not as documented here; upgrade via databricks-core first"
        exit 1
      fi
      ;;
  esac
)

Only after the gate passes, authenticate with the selected profile:

databricks auth describe --profile <PROFILE>

Prefer the latest stable CLI; no online lookup is required. If the gate exits non-zero (older than v1.16.0, missing CLI, or no readable version), or setup-local is absent from help, reports unknown command, or rejects a flag this skill tells you to pass, stop. If it prints NOTE: dev build, the floor is unverifiable -- confirm with the user before continuing. Use databricks-core to upgrade with approval and verify; never recreate setup-local manually.

Use the selected profile for every workspace command. Do not convert another package manager without approval.

2. Confirm the project directory

Confirm the root containing (or intended to contain) pyproject.toml, .venv, and uv.lock; ask if multiple roots are plausible. Use it for preview and apply. It must be greenfield or uv-managed, but need not be writable for preview.

3. Select one target

Choose exactly one branch:

  • Cluster: use --cluster-id <ID> or --cluster-name <NAME>. If unknown, list clusters with the selected profile and ask; see examples.
  • Serverless: use --serverless-version <N>. No version-list command exists; ask if unspecified.
  • Job task: use --job-task <JOB_ID>.<TASK_KEY>. If the task is unknown, run databricks jobs get <JOB_ID> --profile <PROFILE> --output json, present task keys, and ask.
  • Bundle: a project with databricks.yml. Use databricks-dabs to inspect its root and selected target. Omit compute flags only when that target resolves supported classic or serverless compute; otherwise ask. Add --target <BUNDLE_TARGET> for a named target.

Never combine compute flags. If no branch resolves, ask the user.

4. Preview

Dry-run first; it writes and installs nothing:

databricks environments setup-local --profile <PROFILE> <TARGET_ARGS> --dry-run --output json

For bundles, <TARGET_ARGS> is empty or --target <BUNDLE_TARGET>. Default to normal mode. Use --no-dbconnect only when the user explicitly does not want this command managing databricks-connect; the result's mode still reports constraints-only for it. See JSON output and examples.

5. Obtain approval and apply

Before apply, verify the directory is writable and run uv --version. Ask before installing uv; never silently set DATABRICKS_LOCALENV_AUTO_INSTALL_UV=1 or run a remote installer.

Show the target, versions, warnings, plan.diff, and directory. Explain that apply may:

  • back up and rewrite pyproject.toml;
  • install Python and dependencies;
  • update .venv and uv.lock.

For --serverless-version N in a bundle, also disclose the post-apply job YAML synchronization described below so approval covers both mutations.

Apply only after the user requested provisioning or approves that plan for the named directory. Preserve the directory, profile, target, and mode.

If the directory, profile, target, mode, or project files change after preview, rerun --dry-run, show the new plan, and obtain approval again. Treat its resolved Python, databricks-connect, and managed constraints as authoritative; do not substitute guessed versions. Reconcile user-owned dependency conflicts separately, with approval.

6. Handle the result

  • ok: true: for --serverless-version N in a bundle, update every existing job environments[].spec.environment_version in its YAML sources to "N", then validate the bundle. Report if none exist; do not invent one. Skip this for cluster and job-task targets. Report target, versions, warnings, and venvPath; prefer uv run <cmd> or derive the platform-specific interpreter from venvPath.
  • ok: false: use troubleshooting. Ask before diagnostic runs that mutate files and before filing an external issue.
  • No JSON: inspect stderr for a pre-pipeline CLI, authentication, directory, or cache error.

Spark Structured Streaming

Production-ready streaming pipelines with Spark Structured Streaming. This skill provides navigation to detailed patterns and best practices.

Quick Start

from pyspark.sql.functions import col, from_json

# Basic Kafka to Delta streaming
df = (spark
    .readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "broker:9092")
    .option("subscribe", "topic")
    .load()
    .select(from_json(col("value").cast("string"), schema).alias("data"))
    .select("data.*")
)

df.writeStream \
    .format("delta") \
    .outputMode("append") \
    .option("checkpointLocation", "/Volumes/catalog/checkpoints/stream") \
    .trigger(processingTime="30 seconds") \
    .start("/delta/target_table")

Core Patterns

PatternDescriptionReference
Kafka StreamingKafka to Delta, Kafka to Kafka, Real-Time ModeSee references/kafka-streaming.md
Real-Time Mode (RTM)Sub-second E2E latency — cluster setup, slot math, supported ops (incl. stream-stream inner join on DBR 18+), transformWithState, observability, error classes, delivery semanticsSee references/real-time-mode.md
Lakebase SinkWrite streaming records into Lakebase Postgres with transactional upserts. Native format("postgresql") sink (DBR 18.3+) and manual foreach sink as a fallbackSee references/lakebase-sink-python.md
Stream JoinsStream-stream joins, stream-static joinsSee references/stream-stream-joins.md, references/stream-static-joins.md
Multi-Sink WritesWrite to multiple tables, parallel mergesSee references/multi-sink-writes.md
Merge OperationsMERGE performance, parallel merges, optimizationsSee references/merge-operations.md
Custom State (transformWithState)Built-in stateful operator or transformWithState; timers (timeMode="ProcessingTime" never lets AvailableNow jobs finish), TTL, initial state, schema evolution, state reader, migrating from applyInPandasWithStateSee references/transform-with-state.md

Configuration

TopicDescriptionReference
CheckpointsCheckpoint management and best practicesSee references/checkpoint-best-practices.md
Stateful OperationsWatermarks, state stores, RocksDB configurationSee references/stateful-operations.md
Trigger & CostTrigger selection, cost optimization, RTMSee references/trigger-and-cost-optimization.md

Best Practices

TopicDescriptionReference
Production ChecklistComprehensive best practicesSee references/streaming-best-practices.md

Production Checklist

  • Checkpoint location is persistent (UC volumes, not DBFS)
  • Unique checkpoint per stream
  • Fixed-size cluster (no autoscaling for streaming)
  • Monitoring configured (input rate, lag, batch duration)
  • Exactly-once verified (txnVersion/txnAppId)
  • Watermark configured for stateful operations
  • New custom state uses transformWithState where supported, with timeMode="EventTime" or "None" in scheduled or serverless jobs
  • Left joins for stream-static (not inner)

Catalog and schema are always user-supplied — never default to any value. If the user hasn't provided them, ask. For any UC write, always create the schema if it doesn't exist before writing data.

Databricks Synthetic Data Generation

Generate realistic, story-driven synthetic data for Databricks using Spark + Faker + Pandas UDFs (strongly recommended).

Data Must Tell a Business Story

Synthetic data should demonstrate how Databricks helps solve real business problems.

The pattern: Something goes wrong → business impact ($) → analyze root cause → identify affected customers → fix and prevent.

Key principles:

  • Problem → Impact → Analysis → Solution — Include an incident, anomaly, or issue that causes measurable business impact. The data lets you find the root cause and act on it.
  • Industry-relevant but simple — Use domain terms (e.g., "SLA breach", "churn", "stockout") but keep the schema easy to understand. A few tables, clear relationships.
  • Business metrics with $ impact — Revenue, MRR, cost, conversion rate. Every story needs a dollar sign to show why it matters.
  • Tables explain each other — Ticket spike? Incident table shows the outage. Revenue drop? Churn table shows who left and why. All data connects.
  • Actionable insights — Data should answer: What happened? Who's affected? How much did it cost? How do we prevent it?

Why no flat distributions: Uniform data has no story — no spikes, no anomalies, no cohort, no 20/80, no skew, nothing to investigate. It can't show Databricks' value for root cause analysis.

References

WhenGuide
User mentions ML model training or complex time patternsreferences/1-data-patterns.md — ML-ready data, time multipliers, row coherence
Errors during generationreferences/2-troubleshooting.md — Fixing common issues

Critical Rules

  1. Data tells a story — Something goes wrong, impacts $, can be analyzed and fixed. Show Databricks value.
  2. All data serves the story — Every table and column must be coherent and usable in dashboards or ML models. No orphan data, no random noise — if it doesn't help explain or plot a futur dashboard or predict, don't generate it.
  3. Industry terms, simple schema — Use domain-specific vocabulary but keep it easy to understand (few tables, clear relationships)
  4. Never uniform distributions — Skewed categories, log-normal amounts, 80/20 patterns. Flat = no story = useless
  5. Enough data for trends — ~100K+ rows for main tables so patterns survive aggregation
  6. Ask for catalog/schema — Never default, always confirm before generating
  7. Present plan for approval — Show tables, distributions, assumptions before writing code
  8. Master tables first — Generate parent tables, write to Delta, then create children with valid FKs
  9. Use Spark + Faker + Pandas UDFs — Scalable, parallel. Polars only if user explicitly wants local + <30K rows
  10. Use Databricks Connect Serverless by default to generate data — Update databricks-connect on python 3.12 if required (avoid using execute_code unless instructed to not use Databricks Connect)
  11. No .cache() or .persist() — Not supported on serverless. Write to Delta, read back for joins
  12. No Python loops or .collect() — Use Spark parallelism. No driver-side iteration, avoid Pandas↔Spark conversions

Generation Planning Workflow

Before generating any code, you MUST present a plan for user approval.

⚠️ MUST DO: Confirm Catalog Before Proceeding

You MUST explicitly ask the user which catalog to use. Do not assume or proceed without confirmation.

Example prompt to user:

"Which Unity Catalog should I use for this data?"

When presenting your plan, always show the selected catalog prominently:

📍 Output Location: catalog_name.schema_name
   Volume: /Volumes/catalog_name/schema_name/raw_data/

This makes it easy for the user to spot and correct if needed.

Step 1: Gather Requirements

Ask the user about:

  • Catalog/Schema — Which catalog to use?
  • Domain — E-commerce, support tickets, IoT, financial? (Use industry terms)

If user doesn't specify a story: Propose one. Don't generate bland data — suggest an incident, anomaly, or trend that shows Databricks value (e.g., "I'll include a system outage that causes ticket spike and churn — this lets you demo root cause analysis").

Step 2: Present Plan with Story

Show a clear specification with the business story and your assumptions surfaced:

📍 Output Location: {user_catalog}.support_demo
   Volume: /Volumes/{user_catalog}/support_demo/raw_data/

📖 Story: A payment system outage causes support ticket spike. Resolution times
   degrade, enterprise customers churn, revenue drops $2.3M. With Databricks we
   identify the root cause, affected customers, and prevent future impact.
TableDescriptionRowsKey Assumptions
customersCustomer profiles with tier, MRR10,000Enterprise 10% but 60% of revenue
ticketsSupport tickets with priority, resolution_time80,000Spike during outage, SLA breaches
incidentsSystem events (outages, deployments)50Payment outage mid-month
churn_eventsCustomer cancellations with reason500Spike after poor support experience

Business metrics:

  • customers.mrr — Revenue at risk ($)
  • tickets.resolution_hours — SLA performance
  • churn_events.lost_mrr — Churn impact ($)

The story this data tells:

  • Incident table shows payment outage on March 15
  • Tickets spike 5x during outage, resolution time degrades from 4h → 18h
  • Enterprise customers with SLA breaches churn 3 weeks later
  • Total impact: $2.3M lost MRR, traceable to one incident
  • Databricks value: Root cause analysis, identify at-risk customers, build alerting

Ask user: "Does this story work? Any adjustments?"

Step 3: Ask About Data Features

  • Skew (non-uniform distributions) - Enabled by default
  • Joins (referential integrity) - Enabled by default
  • Bad data injection (for data quality testing)
  • Multi-language text
  • Incremental mode (append instead of overwrite)

Pre-Generation Checklist

  • Catalog confirmed - User explicitly approved which catalog to use
  • Output location shown prominently in plan (easy to spot/change)
  • Table specification shown and approved
  • Assumptions about distributions confirmed
  • User confirmed compute preference (Databricks Connect on serverless recommended)
  • Data features selected

Do NOT proceed to code generation until user approves the plan, including the catalog.

Post-Generation Validation

Use databricks experimental aitools tools query to validate generated data (row counts, distributions, referential integrity). Query parquet files directly:

databricks experimental aitools tools query --warehouse $WAREHOUSE_ID "
SELECT COUNT(*) FROM parquet.\`/Volumes/CATALOG/SCHEMA/raw_data/customers\`
"

See references/2-troubleshooting.md for full validation examples.

Use Databricks Connect Spark + Faker Pattern

from databricks.connect import DatabricksSession
from pyspark.sql import functions as F
from pyspark.sql.types import StringType
import pandas as pd

# Setup serverless Spark session
spark = DatabricksSession.builder.serverless(True).getOrCreate()

# Pandas UDF pattern - import lib INSIDE the function (libs must be installed locally)
@F.pandas_udf(StringType())
def fake_name(ids: pd.Series) -> pd.Series:
    from faker import Faker  # Import inside UDF
    fake = Faker()
    return pd.Series([fake.name() for _ in range(len(ids))])

# Generate with spark.range, apply UDFs
customers_df = spark.range(0, 10000, numPartitions=16).select(
    F.concat(F.lit("CUST-"), F.lpad(F.col("id").cast("string"), 5, "0")).alias("customer_id"),
    fake_name(F.col("id")).alias("name"),
)

# Write to Volume as Parquet (default for raw data)
# Path is a folder with table name: /Volumes/catalog/schema/raw_data/customers/
spark.sql(f"CREATE SCHEMA IF NOT EXISTS {CATALOG}.{SCHEMA}")
spark.sql(f"CREATE VOLUME IF NOT EXISTS {CATALOG}.{SCHEMA}.raw_data")
customers_df.write.mode("overwrite").parquet(f"/Volumes/{CATALOG}/{SCHEMA}/raw_data/customers")

Partitions by scale: spark.range(N, numPartitions=P)

  • <100K rows: 8 partitions
  • 100K-500K: 16 partitions
  • 500K-1M: 32 partitions
  • 1M+: 64+ partitions

Output formats:

  • Parquet to Volume (default): df.write.parquet("/Volumes/.../raw_data/table") — raw data for pipelines
  • Delta Table: df.write.saveAsTable("catalog.schema.table") — if user wants queryable tables
  • JSON/CSV: small dimension tables, replicate legacy systems

Performance Rules

Generated scripts must be highly performant. Never do these:

Anti-PatternWhy It's SlowDo This Instead
Python loops on driverSingle-threaded, no parallelismUse spark.range() + Spark operations
.collect() then iterateBrings all data to driver memoryKeep data in Spark, use DataFrame ops
Pandas → Spark → PandasSerialization overhead, defeats distributionStay in Spark, use pandas_udf only for UDFs
Read/write temp filesUnnecessary I/OChain DataFrame transformations
Scalar UDFsRow-by-row processingUse pandas_udf for batch processing

Good pattern: spark.range() → Spark transforms → pandas_udf for Faker → write directly

Common Patterns

Weighted Categories (never uniform)

F.when(F.rand() < 0.6, "Free").when(F.rand() < 0.9, "Pro").otherwise("Enterprise")

Log-Normal Amounts (in a pandas UDF)

Use np.random.lognormal(mean, sigma) — always positive, long tail:

  • Enterprise: lognormal(7.5, 0.8) → ~$1800 median
  • Pro: lognormal(5.5, 0.7) → ~$245 median
  • Free: lognormal(4.0, 0.6) → ~$55 median

Date Range (Last 6 Months)

END_DATE = datetime.now()
START_DATE = END_DATE - timedelta(days=180)

Infrastructure (always create in script)

spark.sql(f"CREATE SCHEMA IF NOT EXISTS {CATALOG}.{SCHEMA}")
spark.sql(f"CREATE VOLUME IF NOT EXISTS {CATALOG}.{SCHEMA}.raw_data")

Referential Integrity (FK pattern)

Write master table to Delta first, then read back for FK joins (no .cache() on serverless):

# 1. Write master table
customers_df.write.mode("overwrite").saveAsTable(f"{CATALOG}.{SCHEMA}.customers")

# 2. Read back for FK lookup
customer_lookup = spark.table(f"{CATALOG}.{SCHEMA}.customers").select("customer_idx", "customer_id")

# 3. Generate child table with valid FKs via join
orders_df = spark.range(N_ORDERS).select(
    (F.abs(F.hash(F.col("id"))) % N_CUSTOMERS).alias("customer_idx")
)
orders_with_fk = orders_df.join(customer_lookup, on="customer_idx")

Setup

Requires Python 3.12 and databricks-connect>=16.4. Use uv:

uv pip install "databricks-connect>=16.4,<17.4" faker numpy pandas holidays

Related Skills

  • databricks-unity-catalog — Managing catalogs, schemas, and volumes
  • databricks-dabs — DABs for production deployment

Common Issues

IssueSolution
ModuleNotFoundError: fakerInstall locally: uv pip install faker, import inside UDF
Faker UDF is slowUse pandas_udf for batch processing
Out of memoryIncrease numPartitions in spark.range()
Referential integrity errorsWrite master table to Delta first, read back for FK joins
PERSIST TABLE is not supported on serverlessNEVER use .cache() or .persist() with serverless - write to Delta table first, then read back
F.window vs Window confusionUse from pyspark.sql.window import Window for row_number(), rank(), etc. F.window is for streaming only.
Broadcast variables not supportedNEVER use spark.sparkContext.broadcast() with serverless

See references/2-troubleshooting.md for full troubleshooting guide.

Unity Catalog

Guidance for Unity Catalog governance — access control, the privilege model, external locations, securable DDL, and fine-grained access — plus system tables and volume file operations.

Before running databricks CLI commands, confirm the CLI and the subcommand exist. Run databricks --version — this skill assumes the unified CLI (≥ v1.0.0). Several subcommands shown here (experimental aitools, system-schemas, external-lineage, grants) vary by version or workspace availability; if one is missing or rejects a flag, fall back to the SQL form or the Python SDK rather than guessing. Each reference notes its own version floor where relevant.

When to Use This Skill

Use this skill when:

Governance & access control (start here):

  • Granting or revoking access — GRANT/REVOKE, the UC privilege model, ownership (ALTER … OWNER TO), SHOW GRANTS, "who can read/write this table?"
  • Row- and column-level security — row filters, column masks, dynamic views with current_user() / is_account_group_member()
  • External locations & storage credentials — CREATE STORAGE CREDENTIAL, CREATE EXTERNAL LOCATION, backing external tables/volumes
  • Securable DDL & metadata — creating/altering catalogs, schemas, managed vs external tables, views; comments, tags, table properties, ownership
  • Connection credentials & secrets: a connection's password from a secret scope (secret()) and its rotation; where the secret lives

Observability & files:

  • Working with volumes (upload, download, list files in /Volumes/)
  • Querying lineage (table dependencies, column-level lineage)
  • Analyzing audit logs (who accessed what, permission changes)
  • Monitoring billing and usage (DBU consumption, cost analysis)
  • Tracking compute resources (cluster usage, warehouse metrics)
  • Reviewing job execution (run history, success rates, failures)
  • Analyzing query performance (slow queries, warehouse utilization)
  • Profiling data quality (data profiling, drift detection, metric tables)

Reference Files

TopicFileDescription
Access Controlreferences/1-access-control.mdPrivilege model, securable hierarchy, GRANT/REVOKE, ownership, inheritance, SHOW GRANTS
External Locationsreferences/2-external-locations.mdStorage credentials (AWS/Azure/GCP), external locations, validation
Securables DDLreferences/3-securables-ddl.mdCREATE/ALTER/DROP catalogs/schemas/tables/views, comments, tags, ownership
Fine-Grained Accessreferences/4-fine-grained-access.mdRow filters, column masks, dynamic views
System Tablesreferences/5-system-tables.mdLineage, audit, billing, compute, jobs, query history
Volumesreferences/6-volumes.mdVolume file operations, permissions, best practices
Data Profilingreferences/7-data-profiling.mdData profiling, drift detection, profile metrics
Connections & Secretsreferences/8-connections-and-secrets.mdA connection's password and its rotation; where the secret lives

Quick Start

Create Unity Catalog Objects (CLI)

Use --json for create commands. Positional argument order differs per command and has changed across CLI versions, so --json is the order-independent, version-stable form shown throughout this skill.

# Create a catalog
databricks catalogs create --json '{"name": "my_catalog"}'

# Create a schema
databricks schemas create --json '{"name": "my_schema", "catalog_name": "my_catalog"}'

# Create a managed volume
databricks volumes create --json '{
  "catalog_name": "my_catalog",
  "schema_name": "my_schema",
  "name": "my_volume",
  "volume_type": "MANAGED"
}'

# List catalogs, schemas, volumes (read commands take simple positional args)
databricks catalogs list
databricks schemas list my_catalog
databricks volumes list my_catalog.my_schema

Positional create args still work if you prefer them, but the order is not uniform across commands — this is the per-command order (and the reason --json is recommended):

CommandPositional create order
databricks catalogs createNAME
databricks schemas createNAME CATALOG_NAME
databricks volumes createCATALOG_NAME SCHEMA_NAME NAME VOLUME_TYPE

CLI surface varies by version. If a databricks subcommand or positional signature is missing in your install, prefer --json, the SQL form, or the Python SDK rather than guessing flags.

Volume File Operations (CLI)

databricks fs requires the dbfs: scheme prefix even for UC Volume paths — without it the CLI treats the path as local filesystem and errors with no such directory.

# List files in a volume
databricks fs ls dbfs:/Volumes/catalog/schema/volume/path/

# Upload a directory's contents to a volume (-r copies contents, not the directory itself)
databricks fs cp -r --overwrite /tmp/data dbfs:/Volumes/catalog/schema/volume/dest

# Download a file from a volume
databricks fs cp dbfs:/Volumes/catalog/schema/volume/file.csv /tmp/file.csv

# Create a directory in a volume
databricks fs mkdirs dbfs:/Volumes/catalog/schema/volume/new_folder

Grant & Revoke Access

GRANT/REVOKE is the core governance operation. See references/1-access-control.md for the full privilege model.

-- Grant read access on a schema to a group
GRANT USE CATALOG ON CATALOG analytics TO `data_readers`;
GRANT USE SCHEMA ON SCHEMA analytics.gold TO `data_readers`;
GRANT SELECT ON SCHEMA analytics.gold TO `data_readers`;

-- Who can access this table?
SHOW GRANTS ON TABLE analytics.gold.customers;

-- Revoke
REVOKE SELECT ON SCHEMA analytics.gold FROM `data_readers`;

Enable System Tables Access

-- Grant access to system tables
GRANT USE CATALOG ON CATALOG system TO `data_engineers`;
GRANT USE SCHEMA ON SCHEMA system.access TO `data_engineers`;
GRANT SELECT ON SCHEMA system.access TO `data_engineers`;

Common Queries

-- Table lineage: What tables feed into this table?
SELECT source_table_full_name, source_column_name
FROM system.access.table_lineage
WHERE target_table_full_name = 'catalog.schema.table'
  AND event_date >= current_date() - 7;

-- Audit: Recent permission changes
SELECT event_time, user_identity.email, action_name, request_params
FROM system.access.audit
WHERE action_name LIKE '%GRANT%' OR action_name LIKE '%REVOKE%'
ORDER BY event_time DESC
LIMIT 100;

-- Billing: DBU usage by workspace
SELECT workspace_id, sku_name, SUM(usage_quantity) AS total_dbus
FROM system.billing.usage
WHERE usage_date >= current_date() - 30
GROUP BY workspace_id, sku_name;

Running SQL from the CLI

databricks experimental aitools tools query is an experimental command. The experimental namespace is not guaranteed to be stable across CLI versions and may be absent in your install. Prefer running system-table SQL from a SQL warehouse (SQL editor, scheduled query) or the Python SDK (w.statement_execution.execute_statement), or a notebook. Use the experimental CLI only for quick ad-hoc checks.

Getting the IDs these examples use. WAREHOUSE_ID — run databricks warehouses list (or copy it from a SQL warehouse's Connection details in the UI). METASTORE_ID (used in references/5-system-tables.md) — w.metastores.current().metastore_id via the SDK, or the Catalog UI → metastore details.

Experimental CLI form (convenience only):

databricks experimental aitools tools query --warehouse WAREHOUSE_ID "
  SELECT source_table_full_name, target_table_full_name
  FROM system.access.table_lineage
  WHERE event_date >= current_date() - 7
"

Stable SDK fallback (works on any CLI version):

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
resp = w.statement_execution.execute_statement(
    warehouse_id="WAREHOUSE_ID",
    statement="""
        SELECT source_table_full_name, target_table_full_name
        FROM system.access.table_lineage
        WHERE event_date >= current_date() - 7
        LIMIT 100
    """,
)
for row in resp.result.data_array or []:
    print(row)

CLI surface varies by version. If a databricks subcommand (e.g. an experimental tool, system-schemas, or external-lineage) is missing, fall back to the SQL warehouse or the Python SDK shown above rather than guessing flags.

Best Practices

  1. Grant minimal access - Apply least privilege; grant at the narrowest securable that works
  2. Filter by date - System tables can be large; always use date filters
  3. Use appropriate retention - Check your workspace's retention settings
  4. Schedule reports - Create scheduled queries for regular monitoring
  5. Prefer SQL/SDK over experimental CLI - For anything beyond quick checks
  6. Connection passwords - after a rotation, run ALTER CONNECTION with every option and the same secret(), or databricks connections update for a connection made without secret(). Azure Key Vault-backed scopes: the vault's access-policy model only, and a vault in another tenant needs the creating user to be able to create service principals there. Where the secret lives: references/8-connections-and-secrets.md

Related Skills

This skill owns Unity Catalog governance: access control, the privilege model, external locations / storage credentials, securable DDL, fine-grained access, system tables, and volumes. For adjacent concerns, use the sibling skill instead:

Roadmap (not yet covered — deferred to a later version)

These governance areas are intentionally out of scope for v0.4.0 and planned for later:

  • Delta Sharing / Marketplace / Clean Rooms
  • Lakehouse Federation (connection setup + foreign catalogs; connection passwords: Best Practice 6)
  • ABAC / governed tags as policy

Resources

Unity Gateway

Use the parent databricks-core skill for CLI authentication and profile selection.

Unity Gateway provides Unity Catalog securables for governed access to AI services. The CLI command group keeps the earlier product name: databricks ai-gateway --help.

Choose the Correct Gateway

RequestUse
Unity Gateway model service, MCP service, or model provider serviceThis skill and databricks ai-gateway
AI Gateway configuration on a Model Serving endpoint, including put-ai-gatewaydatabricks-model-serving and databricks serving-endpoints
Lakeflow Connect ingestion gatewaydatabricks-lakeflow-connect

Treat Unity AI Gateway as an earlier name for Unity Gateway when the resource is a Unity Catalog securable. Do not translate legacy Model Serving AI Gateway resources into Unity Gateway resources unless the user explicitly requests a migration.

CLI Workflow

Use the Databricks CLI for resource lifecycle and permission management. Do not substitute direct REST calls or a Databricks SDK for those operations. Python clients are supported for querying model and model provider services; read Query services.

An explicit request for Unity Gateway, Unity AI Gateway, or databricks ai-gateway belongs to this skill. Do not switch to databricks-model-serving merely because the request involves a model service or a workspace. Use that skill only for a legacy per-endpoint AI Gateway operation or when a provisioned-throughput destination requires Model Serving endpoint details.

  1. Verify the CLI meets the minimum version:

    databricks --version
    
  2. Inspect the command and the relevant operation before constructing a payload:

    databricks ai-gateway --help
    databricks ai-gateway <operation> --help
    
  3. Pass create and update configuration through --json. Use an explicit profile when profile-based authentication is required:

    databricks ai-gateway <operation> --json @payload.json --profile <PROFILE>
    

Do not invent JSON fields from similarly named legacy APIs. Build the payload from the installed CLI help and the Unity Gateway documentation for the selected service type.

Service Types

Service typeCLI operations
Model servicecreate-model-service, get-model-service, list-model-services, update-model-service, delete-model-service; read Model services
MCP servicecreate-mcp-service, get-mcp-service, list-mcp-services, update-mcp-service, delete-mcp-service; read MCP services
Model provider servicecreate-model-provider-service, get-model-provider-service, list-model-provider-services, update-model-provider-service, delete-model-provider-service; read Model provider services

Querying and permissions

Do not fetch public documentation for fields already covered by these references. Consult the authoritative documentation only when a required field is absent from the reference or the user explicitly asks for the latest documentation. Do not assign a Beta or preview status unless the installed CLI help or current documentation explicitly does so.

Migration from Model Serving to Unity Gateway

When migrating from workspace-scoped Model Serving endpoints to Unity Catalog-scoped Unity Gateway services, read references/model-serving-migration.md.

Authoritative Documentation

Unstructured-Document for Demos and Eval Datasets on Databricks

Workflow for producing synthetic PDF documents + paired test questions as a Unity Catalog-resident dataset for Demos and RAG / unstructured-document retrieval evaluation on Databricks. The PDF-generation step uses standard local HTML → PDF tooling; the Databricks-specific value is the workflow shape — UC volume layout, paired question files, and integration with downstream Databricks retrieval / ai_extract / ai_parse_document evaluation.

Workflow

  1. Write HTML files to ./raw_data/html/ (write multiple files in parallel for speed) — domain-shaped to match the documents your retrieval pipeline will see in production.
  2. Convert HTML → PDF using <SKILL_ROOT>/scripts/pdf_generator.py (parallel conversion, wraps plutoprint).
  3. Upload PDFs to a Unity Catalog volume via databricks fs cp — same volume shape your production pipeline will read from.
  4. Generate ./raw_data/pdf/pdf_eval_questions.json pairing each document with retrieval-eval questions; this becomes the gold dataset for mlflow.genai.evaluate() or comparable retrieval-quality scorers.

If you only need ad-hoc PDFs (no Databricks workflow), any HTML → PDF tool (weasyprint, wkhtmltopdf, playwright pdf, plutoprint) works directly — this skill exists for the synthetic-dataset-on-UC end-to-end shape, not as a general PDF generator.

Path convention: <SKILL_ROOT> below = the directory containing this SKILL.md. Resolve to the absolute install path (e.g. ~/.claude/skills/databricks-unstructured-pdf-generation). ./raw_data/... paths are relative to your own project cwd.

Dependencies

uv pip install plutoprint

Step 1: Write HTML Files

mkdir -p ./raw_data/html

Write HTML documents to ./raw_data/html/filename.html. Use subdirectories to organize (structure is preserved).

Step 2: Convert to PDF

# Convert entire folder (parallel, 4 workers)
python <SKILL_ROOT>/scripts/pdf_generator.py convert --input ./raw_data/html --output ./raw_data/pdf

Skips files where PDF exists and is newer than HTML. Use --force to reconvert all.

Step 3: Upload to Volume

databricks fs requires the dbfs: scheme prefix even for UC Volume paths. -r copies the contents of the source directory into the target (the source directory name is not preserved), so name the target raw_data/pdf explicitly to keep the PDFs in their own folder on the volume. They land under raw_data/pdf/ — i.e. dbfs:/Volumes/my_catalog/my_schema/raw_data/pdf/report.pdf — so an ingest pipeline can point at that single folder. For a Genie Agent, upload to a dedicated managed volume instead (e.g. dbfs:/Volumes/my_catalog/my_schema/docs/), because Genie indexes the whole volume, not a subfolder (see databricks-genie-agents § Documents).

databricks fs cp -r --overwrite ./raw_data/pdf dbfs:/Volumes/my_catalog/my_schema/raw_data/pdf

Step 4: Generate Test Questions

Create ./raw_data/pdf/pdf_eval_questions.json with questions for evaluating a Genie Agent (or a legacy Knowledge Assistant) over the PDFs. Keep it out of a volume attached to a Genie Agent: the whole volume is indexed, so the answers would leak into the agent's context (upload the PDFs before writing it, or databricks fs rm it from that volume):

{
  "api_errors_guide.pdf": {
    "question": "What is the solution for error ERR-4521?",
    "expected_fact": "Call /api/v2/auth/refresh with refresh_token before the 3600s TTL expires"
  },
  "installation_manual.pdf": {
    "question": "What port does the service use by default?",
    "expected_fact": "Port 8443 for HTTPS, configurable via CONFIG_PORT environment variable"
  }
}

Use it to build Agent-mode test cases (or Genie benchmarks) and validate retrieval accuracy.

Document Content Guidelines

When generating documents for Genie Agent (document Q&A) testing or demos:

  • Multi-page documents: Each PDF should be several pages with substantial content
  • Specific error codes and solutions: Include product-specific error codes, causes, and resolution steps
  • Technical details: API endpoints, configuration parameters, version numbers, specific commands
  • Simple CSS: Keep styling minimal for fast HTML creation and reliable PDF conversion
  • Queryable facts: Include details the agent must read the document to answer (not general knowledge)

Good document types:

  • Product user manuals with troubleshooting sections
  • API error reference guides (error codes, causes, solutions)
  • Installation/configuration guides with specific steps
  • Technical specifications with version-specific details

Example content: Instead of generic "Connection failed" errors, write:

  • "Error ERR-4521: OAuth token expired. Cause: Token TTL exceeded 3600s default. Solution: Call /api/v2/auth/refresh with your refresh_token before expiration. See Section 4.2 for token lifecycle management."

CLI Reference

python <SKILL_ROOT>/scripts/pdf_generator.py convert [OPTIONS]

  --input, -i     Input HTML file or folder (required)
  --output, -o    Output folder for PDFs (required)
  --force, -f     Force reconvert (ignore timestamps)
  --workers, -w   Parallel workers (default: 4)

Folder Structure

Subfolder structure is preserved:

./raw_data/html/                    ./raw_data/pdf/
├── report.html             →       ├── report.pdf
├── quarterly/                      ├── quarterly/
│   └── q1.html             →       │   └── q1.pdf
└── legal/                          └── legal/
    └── terms.html          →           └── terms.pdf

Bundled Script

This skill ships one helper script:

FileDescription
scripts/pdf_generator.pyHTML → PDF converter (wraps plutoprint); parallel folder conversion with timestamp-skip. Referenced by Step 2 and the CLI Reference.

The script ships at <SKILL_ROOT>/scripts/pdf_generator.py. If it is absent, recreate it from the CLI Reference above (a convert subcommand taking --input/--output/--force/--workers, wrapping plutoprint for HTML → PDF).

Troubleshooting

IssueSolution
"plutoprint not installed"uv pip install plutoprint
PDF looks wrongCheck HTML/CSS syntax
"Volume does not exist"databricks volumes create CATALOG SCHEMA VOLUME_NAME MANAGED (four separate positional args, not catalog.schema.volume)

Databricks AI Search (formerly Vector Search)

FIRST: Use the parent databricks-core skill for CLI basics, authentication, and profile selection.

Patterns for creating, managing, and querying AI Search indexes for RAG and semantic search applications.

Naming: the REST API (/api/2.0/vector-search/...), the databricks vector-search-* CLI groups, and the databricks_vector_search_* Terraform resources keep the Vector Search names (API and tool names); the Python client is databricks-ai-search (AISearchClient).

When to Use

Use this skill when:

  • Building RAG (Retrieval-Augmented Generation) applications
  • Implementing semantic search or similarity matching
  • Creating vector indexes from Delta tables
  • Choosing between storage-optimized and standard endpoints
  • Querying vector indexes with filters

Overview

Databricks AI Search provides managed vector similarity search with automatic embedding generation and Delta Lake integration.

ComponentDescription
EndpointCompute resource hosting indexes (Standard or Storage-Optimized)
IndexVector data structure for similarity search
Delta SyncAuto-syncs with source Delta table
Direct AccessManual CRUD operations on vectors (Standard endpoints only)

Endpoint Types

TypeLatencyCapacityCostBest For
Standard20-50ms320M vectors (768 dim)HigherReal-time, low-latency, high QPS
Storage-Optimized300-500ms1B+ vectors (768 dim)Up to 7x lower10M+ vectors, cost-sensitive

Storage-Optimized supports only TRIGGERED Delta Sync indexes: no CONTINUOUS pipelines, no Direct Access, no target_qps. The endpoint type cannot be changed after creation.

Index Types

TypeEmbeddingsSyncUse Case
Delta Sync (managed)Databricks computesAuto from DeltaEasiest setup
Delta Sync (self-managed)You provideAuto from DeltaCustom embeddings
Direct AccessYou provideManual CRUDReal-time updates

Installation

pip install "databricks-ai-search>=0.79"

In a notebook, use %pip install instead and run dbutils.library.restartPython() afterwards: the package requires mlflow-skinny>=3.10.1, so installing it upgrades mlflow in place.

databricks.ai_search first shipped in databricks-ai-search 0.73. This skill needs 0.79+: from 0.79, get_endpoint, list_endpoints and list_indexes always return snake_case keys, and 0.79 also includes the 0.78 fix that makes concurrent similarity_search calls thread-safe (needed for production traffic). From 0.81, service principal auth no longer requires the all-apis OAuth scope.

Existing databricks-vectorsearch code: on databricks-vectorsearch 0.74+, VectorSearchClient is an alias of AISearchClient, so the calls in this skill work on it once databricks-ai-search>=0.79 is installed alongside it. On 0.73 and older it is a separate class: switch the import to from databricks.ai_search.client import AISearchClient. The old class has no start/stop calls, and before 0.68 no target_qps.

Auth: a bare AISearchClient() uses the notebook's credentials. Elsewhere, pass workspace_url plus a personal_access_token, or service_principal_client_id and service_principal_client_secret (needed for high-QPS traffic; see Production and High QPS).

AISearchClient takes plain strings (endpoint_type="STANDARD", pipeline_type="TRIGGERED"). Don't pass the databricks.sdk.service.vectorsearch enums to it; those are for WorkspaceClient.vector_search_*.

Quick Start

Create Endpoint

from databricks.ai_search.client import AISearchClient

client = AISearchClient()

client.create_endpoint(
    name="my-vs-endpoint",
    endpoint_type="STANDARD"  # or "STORAGE_OPTIMIZED"
)
# Note: Endpoint creation is asynchronous; check status with get_endpoint()

create_endpoint_and_wait(..., timeout=datetime.timedelta(minutes=30)) blocks until the endpoint is ONLINE. It fails fast on OFFLINE (and, from 0.80, STOPPING and STOPPED); in other states it polls until the timeout. On a timeout the endpoint may still be provisioning: check client.get_endpoint(name)["endpoint_status"]["state"] before retrying, and don't create it again.

Create Delta Sync Index (Managed Embeddings)

Standard endpoints need Change Data Feed on the source table:

ALTER TABLE catalog.schema.documents SET TBLPROPERTIES (delta.enableChangeDataFeed = true);
# Source table must have: primary key column + text column
index = client.create_delta_sync_index(
    endpoint_name="my-vs-endpoint",
    source_table_name="catalog.schema.documents",
    index_name="catalog.schema.my_index",
    pipeline_type="TRIGGERED",  # "CONTINUOUS" is Standard-only
    primary_key="id",
    embedding_source_column="content",
    embedding_model_endpoint_name="databricks-gte-large-en"
)

An index on an endpoint with no other active indexes can wait for compute even while the endpoint says ONLINE. This typically takes 5-15 minutes, occasionally longer. Its status.detailed_state is PROVISIONING_ENDPOINT ("pending endpoint provisioning"), and sync() is rejected as not ready. The initial sync starts on its own. This also happens when an existing endpoint has released its compute after all indexes were deleted; deleting and recreating the only index can prolong the wait.

  • From an agent's tool calls (which usually time out within minutes): check once per call with databricks vector-search-indexes get-index <index> -o json. Finish when status.detailed_state contains ONLINE and does not contain FAILED; ready alone is true even for ONLINE_PIPELINE_FAILED. Stop on ONLINE_PIPELINE_FAILED or OFFLINE_FAILED and report status.message and status.failed_status. Between checks, tell the user it is still provisioning. After 45 minutes total, report the current status and that provisioning has not finished; do not recreate the index or leave background sleep loops running.
  • From a notebook, job or long-running script: use the bounded 45-minute wait and failure diagnostics in Step 4 of the end-to-end walkthrough.

Query Index

index = client.get_index(
    endpoint_name="my-vs-endpoint",
    index_name="catalog.schema.my_index"
)

results = index.similarity_search(
    query_text="What is machine learning?",
    columns=["id", "content", "metadata"],
    num_results=5
)

# Returns a dict; each row holds the requested columns, then the similarity score
for row in results["result"]["data_array"]:
    print(row)

Common Patterns

Create Storage-Optimized Endpoint

# For large-scale, cost-effective deployments
client.create_endpoint(
    name="my-storage-endpoint",
    endpoint_type="STORAGE_OPTIMIZED"
)

Delta Sync with Self-Managed Embeddings

# Source table must have: primary key + embedding vector column
index = client.create_delta_sync_index(
    endpoint_name="my-vs-endpoint",
    source_table_name="catalog.schema.documents",
    index_name="catalog.schema.my_index",
    pipeline_type="TRIGGERED",
    primary_key="id",
    embedding_dimension=768,
    embedding_vector_column="embedding"
)

Direct Access Index (Standard endpoints only)

# Create index for manual CRUD
index = client.create_direct_access_index(
    endpoint_name="my-vs-endpoint",
    index_name="catalog.schema.direct_index",
    primary_key="id",
    embedding_dimension=768,
    embedding_vector_column="embedding",
    schema={
        "id": "string",
        "text": "string",
        "embedding": "array<float>",
        "metadata": "string"
    }
)

# Upsert data
index.upsert([
    {"id": "1", "text": "Hello", "embedding": [0.1, 0.2, ...], "metadata": "doc1"},
    {"id": "2", "text": "World", "embedding": [0.3, 0.4, ...], "metadata": "doc2"},
])

# Delete data
index.delete(primary_keys=["1", "2"])

Query with Embedding Vector

# When you have pre-computed query embedding
results = index.similarity_search(
    query_vector=[0.1, 0.2, 0.3, ...],  # Your 768-dim vector
    columns=["id", "text"],
    num_results=10
)

Hybrid Search (Semantic + Keyword)

Hybrid search runs ANN and full-text (keyword) search and merges the results with Reciprocal Rank Fusion (RRF). Use it when queries contain exact terms that must match — SKUs, error codes, proper nouns, or technical terminology — where pure semantic search might miss keyword-specific results. See references/search-modes.md for detailed guidance on choosing between ANN and hybrid search.

# Needs an index with a managed embedding model, so query_text alone is enough
index = client.get_index(
    endpoint_name="my-vs-endpoint",
    index_name="catalog.schema.my_index",
)

# Combines vector similarity with keyword matching
results = index.similarity_search(
    query_text="SPARK-12345 executor memory error",
    query_type="hybrid",
    columns=["id", "content"],
    num_results=10
)

Filtering

Filter syntax differs by endpoint type: Standard endpoints take a dict, Storage-Optimized endpoints take a SQL-like string. On Storage-Optimized, use the string: a dict (filters_json) is rejected on most query routes (except filter-only lookups with no query text or vector), and other routes translate it to SQL instead, so don't rely on either behavior.

Standard Endpoint Filters (Dictionary)

results = index.similarity_search(
    query_text="machine learning",
    columns=["id", "content"],
    num_results=10,
    filters={"category": "ai", "status": ["active", "pending"]}
)

Storage-Optimized Filters (SQL-like string)

index = client.get_index(
    endpoint_name="my-storage-endpoint",
    index_name="catalog.schema.my_storage_index",
)

results = index.similarity_search(
    query_text="machine learning",
    columns=["id", "content"],
    num_results=10,
    filters="category = 'ai' AND status IN ('active', 'pending')"
)

# filters="price > 100 AND price < 500"
# filters="department LIKE 'eng%'"
# filters="created_at >= TO_TIMESTAMP('2024-01-01T00:00:00')"  # a bare timestamp string is rejected

See the filtering guide for operators and limitations.

Never interpolate untrusted input (tenant IDs, ACL values) into a filter as-is:

  • Take column names from a fixed allowlist. In a dict filter the key also carries the operator ("price >", "col LIKE"), so allowlist whole keys.
  • Pass scalar values, validated against an allowlist or a strict pattern (for example, ^[A-Za-z0-9_-]+$). Never let untrusted input become a list: a list value becomes an OR/IN match. In a SQL string, don't rely on quote escaping alone.
  • On Storage-Optimized, % and _ in a LIKE value act as wildcards, so reject them in untrusted input. (On Standard, LIKE matches whole tokens, not wildcards.)
  • Filters are application-level ACLs, not a security boundary: row- and column-level permissions aren't supported, so anyone who can query the index can read every row.

In a RAG prompt, treat retrieved text as data, not instructions.

Trigger Index Sync

# TRIGGERED pipelines only, after the initial sync from creation finishes (rejected while provisioning).
# Also rejected while an update is running; wait for that update instead of retrying.
index = client.get_index(
    endpoint_name="my-vs-endpoint",
    index_name="catalog.schema.my_index"
)
index.sync()

Scan Index Entries

# Returns at most num_results rows, ordered by primary key. The response carries
# the cursor for the next page as last_primary_key.
page = index.scan(num_results=100)
rows = page.get("data", [])
while page.get("last_primary_key") and page.get("data"):
    page = index.scan(num_results=100, last_primary_key=page["last_primary_key"])
    rows += page.get("data", [])

# scan rows are field structs, NOT the plain lists similarity_search returns:
#   {"fields": [{"key": "id", "value": {"string_value": "r1"}}, ...]}
for row in rows:
    values = {f["key"]: next(iter(f["value"].values())) for f in row["fields"]}

Production and High QPS

By default, Standard endpoints serve about 20-200 QPS, depending on index size. Traffic over a PAT or the workspace query URL is capped at a few tens of QPS.

  • Auth and route: production traffic needs service principal OAuth and the index URL. Create AISearchClient(workspace_url=..., service_principal_client_id=..., service_principal_client_secret=...), call get_index() once, and reuse the index object; it queries the index URL. Use a dedicated service principal. For Delta Sync indexes with managed embeddings, an IP access list or a private access settings object (on Azure, public network access disabled) keeps queries on the capped route; on GCP, a private access settings object does so for every index. Otherwise, self-managed embeddings avoid it. See requirements.
    • Pitfalls: with any token passed as personal_access_token (a PAT, or a forwarded user or on-behalf-of token such as the Databricks Apps x-forwarded-access-token), or a bare AISearchClient() that picks up notebook or model-serving credentials, get_index() queries the workspace URL even though describe() lists status.index_url; if index.index_url starts with your workspace URL, queries are on the capped route. In Databricks Apps, send query traffic with the app's service principal, not the forwarded user token. workspace_url must include https://: in Databricks Apps, DATABRICKS_HOST can lack the scheme, so add it (example).
  • Query type: start with ANN; hybrid uses about 4x the resources. Load-test with the query type production sends.
  • 429s: move to service principal OAuth and the index URL; wait for SCALING_CHANGE_APPLIED; with query_vector, lower concurrency or raise target_qps; with query_text on a managed-embedding index, the embedding endpoint may be the bottleneck. similarity_search already retries (total_retries, backoff_factor, backoff_jitter); raise those or add your own bounded backoff with jitter. See Troubleshoot 429 errors.
  • Load testing: use Run load test on the endpoint page or the load test notebook. With databricks-ai-search 0.82+, databricks-ai-search guide --topic high-qps prints the guide and databricks-ai-search load-test --index <name> plans a run (it runs only with --yes); skip this if the databricks-ai-search command isn't available.
  • Agent safety: ask before creating endpoints or indexes, setting target_qps, running load tests, or upserting/deleting data, since each costs money or changes data. Never print, paste, or commit tokens or client secrets; read them from a secret scope or environment variables.

Target QPS is Standard-only and best-effort, and the capacity is billed whether or not traffic arrives. Storage-Optimized endpoints reject target_qps. A change can be rejected with RESOURCE_CONFLICT while a previous change is still applying. Nothing is saved, so requested_target_qps still shows the earlier target: poll until scaling_info.state is SCALING_CHANGE_APPLIED (whatever the target), or until scaling_info.target_qps_change_allowed_after_timestamp if set (epoch ms; int() then /1000), otherwise the minutes the error gives, then call update_endpoint again before running the loop below.

import time
from datetime import datetime, timezone

client.update_endpoint(name="my-vs-endpoint", target_qps=500)
# Or at creation: create_endpoint(..., target_qps=500).
# Create the index before this loop: target QPS may not apply until the endpoint has an index.

# Match the requested target too: right after the update, state can still be APPLIED from an earlier target.
# Target QPS is best-effort: if it's still in progress at the deadline, move on.
deadline = time.monotonic() + 30 * 60
while time.monotonic() < deadline:
    info = client.get_endpoint("my-vs-endpoint").get("scaling_info") or {}
    # int(): proto3 JSON may serialize int64 fields as strings
    if info.get("state") == "SCALING_CHANGE_APPLIED" and int(info.get("requested_target_qps") or 0) == 500:
        break
    progress = {k: v for k, v in info.items() if k in ("state", "phase", "index_initialization_percentage", "delayed_reason")}
    if info.get("estimated_completion_timestamp"):
        eta = datetime.fromtimestamp(int(info["estimated_completion_timestamp"]) / 1000, tz=timezone.utc)
        progress["eta"] = eta.isoformat()
    print(progress)
    time.sleep(30)
print(info)

While state is SCALING_CHANGE_IN_PROGRESS, scaling_info also reports progress (Beta; every field is optional and can hold other values, so check each one and report unknown values as-is):

  • estimated_completion_timestamp: expected completion, epoch ms (int() then /1000). It can change as scaling progresses, and it's unset when no estimate is available yet or when delayed_reason is set.
  • phase: ESTIMATING, ADJUSTING_CAPACITY, or INITIALIZING_INDEX (with index_initialization_percentage, 0-100).
  • delayed_reason: CAPACITY_UNAVAILABLE, SCALE_DOWN_BLOCKED, or TAKING_LONGER_THAN_EXPECTED.

Report the ETA, phase, or delay reason to the user instead of guessing, and keep polling until SCALING_CHANGE_APPLIED or the loop's deadline. If all three are unset, that isn't a hang (some endpoints don't report progress): check the endpoint has an index and keep polling state.

ONLINE doesn't mean scaled. These are separate waits:

  • Endpoint provisioned: endpoint_status.state is ONLINE (what create_endpoint_and_wait waits for). ONLINE doesn't mean the target QPS is applied.
  • Target QPS applied: scaling_info.state is SCALING_CHANGE_APPLIED for the requested target (the loop above, run after the index is created). This takes minutes to hours, depending on index size and how much the target changes.
  • Index ready: wait as in Create Delta Sync Index. After PROVISIONING_ENDPOINT, a Delta Sync index typically moves through PROVISIONING_INDEX, PROVISIONING_PIPELINE_RESOURCES, and PROVISIONING_INITIAL_SNAPSHOT; a Direct Access index goes from PROVISIONING_INDEX to ONLINE_DIRECT_ACCESS. A Direct Access index can stay in PROVISIONING_INDEX with a message that creation has failed; stop and report it.

Don't create a second endpoint to get around either wait: it repeats endpoint and index provisioning, and without target_qps it lacks the target capacity.

Load-test only after all three; a load test before SCALING_CHANGE_APPLIED doesn't reflect the target capacity. If the loop above hits its deadline, report that target QPS is still applying instead of load testing.

Search Quality Evaluation (Beta)

Retrieval quality evaluation generates queries from a managed Delta Sync index, runs them with ANN, hybrid and full-text (each with and without the reranker), scores results with an LLM judge, and reports DCG@10 and other metrics.

  • Start it from the index page in Catalog Explorer. In the Search Quality Evaluation section, click Evaluate search quality (or Start new evaluation / Create evaluation under Latest evaluation runs after a run). The built-in Beta evaluation is not configurable. There is no CLI command, SDK method or documented REST API for this built-in evaluation, so don't probe for one. If you can't use the browser, finish the index, then give the user the index page link and these steps, and pick up again from the results.
  • Requirements: a workspace admin turns on AI Search: Quality Evaluation from the workspace Previews page; the person starting a run needs SELECT on the index and may also need CREATE TABLE on its schema, because evaluation writes output tables there. Each run is a job that costs money; its status shows on the index page.
  • Reading results: the index page shows the latest runs and their DCG@10; View results opens the dashboard, and What do my results mean? asks Genie Code to interpret the run. Judged metrics only see the text you indexed: if users search on attributes that live in other columns or tables, also check results against those structured fields, since the judge can't see what's missing.

Reference Files

TopicFileDescription
Index Typesreferences/index-types.mdDetailed comparison of Delta Sync (managed/self-managed) vs Direct Access
End-to-End RAGreferences/end-to-end-rag.mdComplete walkthrough: source table → endpoint → index → query → agent integration
Search Modesreferences/search-modes.mdWhen to use semantic (ANN) vs hybrid search, decision guide
Operationsreferences/troubleshooting-and-operations.mdMonitoring, cost optimization, capacity planning, migration

CLI Quick Reference

# List endpoints
databricks vector-search-endpoints list-endpoints

# Create endpoint (positional args: NAME ENDPOINT_TYPE)
databricks vector-search-endpoints create-endpoint my-endpoint STANDARD

# List indexes on endpoint (positional arg: ENDPOINT_NAME)
databricks vector-search-indexes list-indexes my-endpoint

# Get index status (positional arg: INDEX_NAME)
databricks vector-search-indexes get-index catalog.schema.my_index

# Sync index (positional arg: INDEX_NAME)
databricks vector-search-indexes sync-index catalog.schema.my_index

# Delete index (positional arg: INDEX_NAME)
databricks vector-search-indexes delete-index catalog.schema.my_index

Common Issues

IssueSolution
Index sync slowUse Storage-Optimized endpoints (10-20x faster indexing)
Query latency highUse Standard endpoint for <100ms latency
Filters not workingStandard endpoints use a dict (filters={"col": "val"}); Storage-Optimized use a SQL string (filters="col = 'val'")
Embedding dimension mismatchEnsure query and index dimensions match
Index not updatingCheck pipeline_type; use index.sync() for TRIGGERED. If it's rejected as already running, wait for the current update to finish
Index "pending endpoint provisioning" / sync() "not ready"Normal for an index on an endpoint with no other active indexes, even while the endpoint says ONLINE; wait (see Create Delta Sync Index) instead of recreating it
Evaluating search qualityStart it from the index page; there's no CLI or SDK command. See Search Quality Evaluation
Out of capacityCapacity is sized from the source table at index creation. Load the full table first; past ~320M vectors, recreate on Storage-Optimized (1B+ vectors)
Rate limited (429 errors)See Production and High QPS
query_vector truncatedLarge vectors (e.g. 1024-dim) can be truncated when serialized as JSON. Use query_text instead (for managed embedding indexes), or pass the vector directly via index.similarity_search(query_vector=...)

Embedding Models

Databricks provides built-in embedding models:

ModelDimensionsContext WindowUse Case
databricks-qwen3-embedding-0-6b1024 (configurable, up to 1024)~32K tokensRecommended for production by the AI Search docs (Public Preview)
databricks-gte-large-en10248192 tokensEnglish text, high quality
databricks-bge-large-en1024512 tokensEnglish text, general purpose
# Use with managed embeddings
index = client.create_delta_sync_index(
    ...,
    embedding_source_column="content",
    embedding_model_endpoint_name="databricks-qwen3-embedding-0-6b"
)

Notes

  • Storage-Optimized is newer — better for most use cases unless you need <100ms latency, high QPS, Direct Access, or CONTINUOUS sync
  • Delta Sync recommended — easier than Direct Access for most scenarios
  • Hybrid search — available for both Delta Sync and Direct Access indexes
  • columns_to_sync — limits which columns are synced and returned; only synced columns are available in query results. On Storage-Optimized, the docs' limitations list says it's unsupported, but their full-text example uses it; prefer syncing all columns there
  • Filter syntax differs by endpoint — Standard uses a dict, Storage-Optimized uses a SQL-like string
  • Management vs runtime — CLI and SDK handle lifecycle management; for agent tool-calling at runtime, use VectorSearchRetrieverTool from databricks_langchain or databricks_openai

Related Skills

  • databricks-model-serving - Deploy agents that use VectorSearchRetrieverTool
  • databricks-ml-training - Author and deploy a ResponsesAgent that uses an AI Search retriever
  • databricks-genie-agents - No-code document Q&A: a Genie Agent with a UC volume (replaces the legacy Knowledge Assistant); hand-roll AI Search only for custom retrieval
  • databricks-unstructured-pdf-generation - Generate documents to index in AI Search
  • databricks-unity-catalog - Manage the catalogs and tables that back Delta Sync indexes
  • databricks-pipelines - Build Delta tables used as AI Search sources

Zerobus Ingest

Zerobus Ingest (part of Lakeflow Connect) is a managed, push-based ingestion service: a producer sends records and they land in a Unity Catalog managed Delta table, with no broker or ingestion cluster to run. The SDKs do real work for you: OAuth, buffering, offset tracking, durability acknowledgments, and automatic recovery that replays unacknowledged records after transient failures.

FIRST: Use the parent databricks-core skill for CLI install, authentication, and profile selection.

Decide before you build

Recommend Zerobus when data is pushed from an app, service, or device; the destination is a Delta table; the workload is continuous or event-driven and needs a shock absorber (Zerobus lands data in about 5 seconds, and it still fits when minutes are acceptable); and there is one main downstream consumer (or a message bus such as Apache Kafka, Amazon Kinesis, Azure Event Hubs, or Google Cloud Pub/Sub is just a pipe to the lakehouse).

Don't recommend Zerobus when many independent consumers outside the lakehouse need the data (use a message bus such as Apache Kafka, Amazon Kinesis, Azure Event Hubs, or Google Cloud Pub/Sub), or when the data is already in Kafka or another bus and must be pulled into the lakehouse (use the Lakeflow Connect managed Kafka connector or Spark Structured Streaming). Explain why and name the alternative. Before ruling Zerobus out for files, SaaS apps, or databases, check references/choose-an-interface.md: batches of columnar data can go through Arrow Flight, and a SaaS app that can push to a Zerobus-compatible API can use Zerobus.

Prefer an off-the-shelf path. If the source already speaks OpenTelemetry, Kafka, or MQTT, or runs Debezium, Vector, Telegraf, or Cribl, point the user to that path instead of writing SDK code. All of these, plus the full decision tree, are in references/choose-an-interface.md. For existing Kafka producers, read references/kafka.md; for MQTT v5 devices, references/mqtt.md; for columnar batches (Parquet, Arrow, DataFrames), references/arrow-flight.md; for migrating a pipeline from Kafka or a Kafka-compatible system, read references/migration-from-kafka-compatible-systems.md.

Workflow

Treat the request as a conversation: ask one question at a time, use a multiple-choice question tool if one is available, and always offer "Not sure, help me decide". Skip any question the user has already answered.

Many users won't know Databricks. If the user is an app, DevOps, or platform engineer who's new to Databricks, explain each Databricks term in plain language the first time you use it, map it to something they know (a table name is like database.schema.table in Postgres; a service principal is like a service account), and offer to set up what's missing for them. Read references/new-to-databricks.md.

Step 0: Show your plan

List the steps you'll run and what you'll create (table, grants, producer file). Ask before creating any catalog, schema, table, or service principal; never assume the user wants a new one.

Step 1: Ask where the producer runs

Where will the producer that sends data to Zerobus run?

  • Outside Databricks: a service, app, device or gateway, or my laptop
  • Inside Databricks: a Lakeflow Job or a Databricks App
  • Not sure, help me decide
  • Outside Databricks (most common): write the producer as a local file the user keeps; read credentials from environment variables.
  • Inside Databricks: read references/databricks-apps.md. The Python SDK runs on classic and serverless compute; install it as a job, cluster, or environment dependency.
  • Not sure: ask what generates the data. Data almost always originates outside Databricks, so default there.

Step 2: Ask only the questions you still need

Questions come in three groups. Ask the required ones first: they're all you need to build a first client. Ask the advanced ones only when the user raises the topic, or when the answer would change your recommendation or the deployment.

Required to build a client:

  1. What is the data, and where does it come from? Application events, logs, metrics, traces, CDC from a database, IoT or sensor readings, clickstream? This decides the route: traces, logs, and metrics often fit OTLP, database changes fit Debezium, and an existing Kafka producer fits the Kafka-compatible APIs (references/choose-an-interface.md).
  2. Target table: the three-part name, whether it exists yet, and what shape the records have (a sample event helps). Must any fields always be present, and will the shape change over time (references/schema-management.md)?
  3. Language and runtime: and can the environment use precompiled binaries / native libraries? The Python, TypeScript, Java, Go (cgo), C++, and .NET SDKs load a prebuilt Rust core. If the user doesn't know, assume precompiled binaries are allowed, and change course only if the SDK fails to install or load. If the platform has no prebuilt binary (Alpine, glibc older than 2.34 for Python), or the user wants to build the SDK themselves, build it from source following that SDK README's build steps (references/build-from-source.md). If precompiled binaries aren't allowed, use a pure SDK: Rust or pure Go (pure C doesn't send data yet). Edge runtimes such as Cloudflare Workers or Deno Deploy can't load native code, so use REST there (references/rest.md).

If the user doesn't name a language, default to Python.

Advanced: is Zerobus the right service? Ask when the fit is in doubt, for example when other systems also read the data or the user mentions strict volume or freshness needs (references/choose-an-interface.md):

  • Who consumes it? Is the lakehouse the one main destination, or do several independent systems read the same data? Many consumers outside the lakehouse, replay, or consumer groups mean keep the message bus (don't recommend Zerobus as its replacement).
  • Volume and latency: rough records per second and how fresh the data must be. Zerobus makes records durable in about 150 ms and queryable in about 5 seconds, and it still fits when minutes are acceptable. Volume decides how many streams to open later (references/performance.md); it doesn't change how you build the first client.

Advanced setup: what it takes to run in the user's environment. Ask before deploying beyond a laptop, or when a connection or auth step fails:

Step 3: Confirm prerequisites

Details are in references/authentication.md. Don't run a producer until every row passes. If something is missing, offer to create it: design the table from a sample event, create the service principal, and grant it, one approved step at a time (references/new-to-databricks.md).

#PrerequisiteVerify
1Workspace in a supported regionRegion is listed under availability on the quotas page
2Zerobus endpoint, e.g. https://<workspace-id>.zerobus.<region>.cloud.databricks.com (AWS)Uses the numeric workspace ID and the workspace's region
3A pre-created managed Delta table. Zerobus never creates or alters tablesdatabricks experimental aitools tools discover-schema <catalog.schema.table> --profile <PROFILE> shows the expected columns
4A service principal with an OAuth client ID and secretdatabricks service-principals list --profile <PROFILE> lists it
5Grants: USE CATALOG, USE SCHEMA, and MODIFY + SELECT directly on the table. ALL PRIVILEGES is not sufficient; grant these four explicitlySHOW GRANTS ON TABLE <catalog.schema.table> lists the service principal with MODIFY and SELECT
6Outbound HTTPS (443) from the producer to the endpoint and the workspaceConnectivity check in references/networking.md

Step 4: Build the producer

Read the SDK reference for the chosen language. For production, prefer Protobuf over JSON (references/protobuf-schema.md). Never hard-code secrets.

Always write a non-blocking producer. Use this ladder, and only go further down when the user needs it:

NeedUseBlocks?
Send records (the default)ingest_record_offset() / ingest_records_offset(): queues and returns; the SDK sends and tracks acks in the backgroundNo
Act while ingestion runs: metrics, logging, progress, errors, checkpointsAn ack callback (on_ack / on_error)No
Durability boundary: end of a bounded batch, before shutdownflush() once, then close()Yes, once
Advanced: must not continue until one specific record is durablewait_for_offset(offset)Yes. Never per record or in a loop

Ack callbacks exist in Python, Java, Rust, C++, and pure Go. The TypeScript, CGO Go, and .NET SDKs don't document one; there, call flush() on an interval for progress. Method names differ by language; use the SDK reference.

Minimal Python producer (APIs from the SDK README), with an ack callback for progress and errors:

import os
from zerobus.sdk.sync import ZerobusSdk
from zerobus.sdk.shared import AckCallback, StreamConfigurationOptions, TableProperties

class Progress(AckCallback):
    def on_ack(self, offset: int):  # runs on an SDK background thread; keep it fast
        print(f"durable through offset {offset}")

    def on_error(self, offset: int, error_message: str):
        print(f"error at offset {offset}: {error_message}")

sdk = ZerobusSdk(os.environ["ZEROBUS_SERVER_ENDPOINT"], os.environ["DATABRICKS_WORKSPACE_URL"])
stream = sdk.create_stream(
    os.environ["DATABRICKS_CLIENT_ID"],
    os.environ["DATABRICKS_CLIENT_SECRET"],
    TableProperties(os.environ["ZEROBUS_TABLE_NAME"]),
    StreamConfigurationOptions(ack_callback=Progress()),
)
try:
    for i in range(100):
        stream.ingest_record_offset({"device_name": f"sensor-{i % 10}", "temp": 20 + (i % 15)})  # queues, returns
    stream.flush()  # durability boundary: blocks once until everything queued is acknowledged
finally:
    stream.close()

Step 5: Send a small test batch

Send 10 to 100 records first. A durability acknowledgment (an on_ack callback, or flush() returning) means Zerobus stored the records safely. It does not mean they are queryable yet.

Step 6: Verify the rows landed

Wait about 5 seconds after the acknowledgment (rows become queryable about 5 seconds after they're sent), then run:

databricks experimental aitools tools query \
  "SELECT count(*) AS row_count, max(<timestamp_column>) AS latest FROM <catalog.schema.table>" \
  --profile <PROFILE>

Expected: row_count went up by the number of records you sent, and latest is close to the send time. If not, read references/troubleshooting.md.

Step 7: Harden for production

Before the user scales up, cover schema evolution, duplicates, throughput, networking, and monitoring using the reference map below.

Key facts

  • Table first. Update the table schema before you update producers. Fields that don't fit can go to a rescue column (Beta, JSON only).
  • Durable is not queryable. Zerobus materializes acknowledged data into the Delta table as a separate step: durable in about 150 ms, queryable in about 5 seconds (how-it-works.md).
  • At-least-once delivery. Recovery or a client retry can resend records, so a few duplicates are possible. Most workloads accept them; don't add deduplication by default, and never claim exactly-once. If the user needs unique rows, see how-it-works.md.
  • Ordering is per stream only. No global ordering across streams (how-it-works.md).
  • Non-blocking by default (the Step 4 ladder). Ingest calls block only on backpressure: records wait in an in-memory in-flight buffer until acked, and calls block only when it's full (references/how-it-works.md).
  • Recovery is built in. The SDK reconnects and replays unacked records. Don't hand-roll a retry loop; custom recovery is advanced (references/recovery.md).
  • TIMESTAMP columns take epoch microseconds (an integer), not strings.
  • Quotas are adjustable defaults, set per stream and per table. Add streams before asking for an increase (performance.md).

Common issues

Match the symptom in references/troubleshooting.md (connection, auth and grants, schema, throughput, visibility, duplicates, install / native binaries). Most common: error 4024 = grants not directly on the table; REST 401 after about an hour = expired token.

SDKs: record formats and working examples

Record formats per SDK (JSON, Protobuf, Avro, Arrow Flight) and each SDK's working examples folder: choose-an-interface.md § SDKs. Open its examples before writing code for a specific API. Don't attach a release status to an SDK; use that table. Pure C doesn't send data yet.

Reference map: read only what the task needs

These references hold the decisions, gotchas, and tested minimal samples for each route. For the full API surface, other record types, and the latest SDK versions, fetch the SDK README linked in each file's Sources (raw Markdown); for working code, use the examples folders listed in choose-an-interface.md. docs.databricks.com pages are HTML. Long runnable programs for MQTT and the raw gRPC API are in examples/.

ReadWhen
new-to-databricks.mdThe user doesn't know Databricks: plain-language glossary, and a guided first-time setup (pick a catalog and schema, design the table from a sample event, create the service principal, grant, save settings)
choose-an-interface.mdUnsure whether to use Zerobus, the SDK, REST, or a protocol/tool path; checking runtime compatibility
how-it-works.mdHow Zerobus works: streams, the data path (the async ack loop, then batch commits to Delta), durable vs queryable, how the SDK processes records (in-flight buffer, cumulative acks, backpressure), ordering, delivery and duplicates, scaling
recovery.mdStream failures, retryable vs fatal errors, advanced custom recovery (get_unacked_records, recreate_stream), durable-fallback reprocessing
authentication.mdEndpoints per cloud, service principal OAuth (official), Entra ID / identity federation (SDK-supported, docs pending), grants, env vars, local vs deployed credentials, table-scoped tokens for REST, Kafka, MQTT, and gRPC (mint and refresh), auth errors
python-sdk.mdPython producers: install and platforms, a Protobuf producer, ack callbacks; links to the README for async and other APIs
typescript-sdk.mdNode.js producers and native-binary requirements
go-sdk.mdGo producers with cgo (go/) or pure Go (purego/)
java-sdk.mdJVM producers
other-sdks.mdRust, C++, C#/.NET
kafka.mdExisting Kafka producers: Apache Kafka-compatible producer APIs (Beta), connection settings, token callback, limits, errors
mqtt.mdMQTT v5 devices and gateways: connection settings, CONNECT auth, limits, errors; a runnable non-blocking QoS 1 publisher with token refresh on reconnect in examples/mqtt_producer.py
arrow-flight.mdColumnar batches (Parquet files, Arrow, pandas or Polars DataFrames): schema matching, a non-blocking Python producer, links for other languages, tuning, recovery
migration-from-kafka-compatible-systems.mdMigrating a pipeline from Kafka or a Kafka-compatible system into Delta: when to migrate, the four pathways, step-by-step cutover, rollback
build-from-source.mdThe user wants to build the SDK themselves, or there's no prebuilt binary (Alpine/musl, older glibc, other CPUs): which platforms each SDK ships, each SDK README's source-build steps (Python wheel via make build, TypeScript, Java with JNI, Go, .NET, C++, Rust), an Alpine Dockerfile, build errors
grpc-api.mdBuilding a client from scratch against the gRPC API: the .proto, stub generation, the EphemeralStream messages, auth headers, the async ack loop, recovery; a runnable Python client in examples/grpc_client.py
rest.mdEdge runtimes and serverless functions outside Databricks (Cloudflare Workers, Deno Deploy), languages without an SDK, low-volume senders. Uses a table-scoped token from authentication.md (tokens expire after about an hour)
protobuf-schema.mdGenerating .proto from a table, type mappings
schema-management.mdThe table as the contract: how records are matched, the three strictness scenarios, the rescue column (setup, tag, routing rules), safe schema evolution, Protobuf rules
performance.mdBatching, stream count, quotas, table layout
networking.mdPorts and TLS per interface, front-end PrivateLink, on-prem producers and HTTP proxies, firewalled storage, cross-region, connectivity checks
observability.mdThe monitoring dashboard bundle (install), ack callbacks, system tables, latency (ack and end-to-end freshness), errors, alerts
troubleshooting.mdAny error or missing data
databricks-apps.mdRunning the producer in a Databricks App or Lakeflow Job

Related skills

  • databricks-core: CLI, profiles, auth, and ad hoc SQL for verification
  • databricks-unity-catalog: catalogs, schemas, tables, and grants
  • databricks-pipelines: downstream processing of ingested data, including a deduplicated table built with AUTO CDC
  • databricks-lakeflow-connect-saas: managed connectors when Databricks should pull from a SaaS app, Kafka or RabbitMQ
  • databricks-lakeflow-connect: managed connectors when Databricks should pull from a database
  • databricks-apps / databricks-apps-python: building the Databricks App that hosts a producer
  • databricks-dabs: deploying a producer job with Declarative Automation Bundles
  • databricks-aibi-dashboards: building a dashboard over the ingested events or telemetry

Commands 2

Databricks Doctor

Run a read-only health check and report a short status table. Make no changes; every step below only reads. If a subcommand or flag is unfamiliar, check databricks <group> --help first rather than guessing.

Run these in order. Don't stop on the first failure; collect what you can and report the rest as unknown.

  1. CLI: databricks --version. Flag only if it's missing; don't gate on a specific version (the CLI surfaces its own update notice).
  2. Profiles: databricks auth profiles. List configured profiles and validity. If $1 is given, use that profile for the rest. Otherwise, if more than one profile exists, ask the user which to use (never auto-select).
  3. Auth method: databricks auth describe --profile <profile> shows the effective host, user, and credential source (never pass --sensitive).
  4. Auth validity: databricks current-user me --profile <profile>. This single API call proves the credentials work end to end (token valid, workspace reachable, expected identity); don't probe other APIs for it. For account-level profiles (an accounts.* host), current-user me does not exist; report what auth describe resolved instead.

Stop here by default. Run the extended checks below only when the user passed full or asked about compute or jobs:

  1. Compute: databricks warehouses list and databricks clusters list for the profile. Note what's running.
  2. Recent job failures: list recent job runs (e.g. databricks jobs list-runs --limit 20 --profile <profile>) and surface any recent failures.

Then print a compact table: check | status (✅/⚠️/❌) | detail. End with the single most useful next action (e.g. "run /databricks:setup to add a profile").

This is a status check; it only reads, so don't run anything that changes state.

Databricks Setup

Guide the user through Databricks CLI authentication. Use the databricks-core skill for the authoritative auth details; this command is the step-by-step wrapper around it.

  1. CLI present? databricks --version. If it's missing, follow the install steps in the databricks-core skill (databricks-cli-install.md). In sandboxed environments (Cursor, containers), print the install command and ask the user to run it in their own terminal. Don't try to install into the sandbox.
  2. Existing profiles? databricks auth profiles. Show what's already configured. If a working profile exists, ask whether to reuse it or add a new one. If no profiles exist and the user has no Databricks account yet, don't search the web for signup options. Point them to the free trial at https://www.databricks.com/signup/express?provider=DB&dbx_source=agent-skills&dbxSource=agent-skills&utm_source=agent-skills&utm_campaign=setup (startups can also apply for credits at https://www.databricks.com/product/startups), wait for them to paste the new workspace URL, then continue at step 3 with that URL.
  3. Pick an auth method (ask the user; $1 may be a workspace or account console URL):
    • OAuth U2M (default, interactive): databricks auth login --host <workspace-url> --profile <name>. Opens a browser. Best for laptops. If the user doesn't know their workspace URL, plain databricks auth login --profile <name> opens login.databricks.com to sign in and pick a workspace. URLs copied from the browser may carry ?w=<workspace-id> or account_id= query params; the CLI accepts them, but quote the URL so the shell doesn't interpret the ?.
    • Account-level: when the host is an account console URL (accounts.cloud.databricks.com, accounts.azuredatabricks.net, accounts.gcp.databricks.com), also pass the account ID: databricks auth login --host <account-url> --account-id <uuid> --profile <name>. Ask for the account ID if it isn't in the URL (it's the UUID shown in the account console address bar).
    • PAT: databricks configure --token --profile <name>; the user pastes a personal access token. This command prompts on stdin, so don't run it yourself (it hangs without a TTY): ask the user to run it in their own terminal, then continue once it's done. The same applies to databricks auth login when no browser can open (headless or sandboxed sessions).
    • Service principal (M2M): client id/secret via profile or env. Use for CI/automation; never a personal PAT in CI.
    • In-platform (notebook/cluster): DATABRICKS_HOST/DATABRICKS_TOKEN are already injected, so no setup is needed.
  4. Confirm before writing any profile; auth writes to ~/.databrickscfg.
  5. Verify: databricks current-user me --profile <name> returns the expected user. For account-level profiles, current-user me doesn't exist; use databricks auth describe --profile <name> and check the resolved host and account ID.

Never echo tokens or secrets back. Never auto-select a profile. When done, suggest /databricks:doctor for a full health check.

Sponsors

Details

Marketplace
databricks/databricks-agent-skills
Source
plugins/databricks/claude
Category
Development