apache avatar

banyandb-bydbql

Generate, validate, and execute read-only BanyanDB BydbQL for SkyWalking data.

GitHub stars
355

Install

/plugin install banyandb-bydbql@banyandb

Run it inside Claude Code after adding the marketplace with /plugin marketplace add apache/skywalking-banyandb.

Contents

Discovered from source

Skills 5

BanyanDB BydbQL

Use this skill when the user asks to query BanyanDB, generate BydbQL, translate natural language to BydbQL, inspect BanyanDB data, validate BydbQL, or run read-only BanyanDB queries.

Workflow

  1. Read references/safety.md before generating, validating, or executing BydbQL.
  2. Read references/syntax.md when query syntax is needed.
  3. Read references/examples.md when mapping natural language to BydbQL.
  4. Identify the BanyanDB resource type: STREAM, MEASURE, TRACE, or PROPERTY.
  5. If the group, resource name, or resource type is missing or ambiguous, use list_groups_schemas.
  6. If the request involves sorting or ranking (ORDER BY, SHOW TOP, or wording such as "latest", "top", "highest", "lowest", "first", "last"), or you have already fetched schema, call get_generate_bydbql_prompt and follow its index-rule guidance. This is required: get_generate_bydbql_prompt is the only component that injects the indexed-field list and enforces ORDER BY field substitution/omission. validate_bydbql is parse-only, so ORDER BY <non-indexed-field> validates true but fails at execution.
  7. Generate exactly one read-only BydbQL statement. When it contains ORDER BY, only sort by a field from the indexed-field list returned by get_generate_bydbql_prompt; substitute the closest indexed field, or omit ORDER BY when no indexed field matches.
  8. Call validate_bydbql before execution.
  9. Execute with list_resources_bydbql only when the user asks to run, query, fetch, show, or inspect raw BanyanDB data.
  10. If validate_bydbql returns valid=false, fix the query and validate again before any execution.
  11. If execution fails due to a missing group, resource, or schema, use list_groups_schemas to discover the correct names and retry only when the correction is clear.

Tool Use

Use the existing BanyanDB MCP tools:

  • list_groups_schemas: discover groups and resource names for streams, measures, traces, and properties.
  • get_generate_bydbql_prompt: return the generation prompt text, including the live indexed-field list and the ORDER BY index-rule substitution/omission rules. It is the only source of index-rule ORDER BY safety.
  • validate_bydbql: run read-only safety checks and parse-only BydbQL syntax validation. This does not verify group, resource, tag, field, or index-rule existence.
  • list_resources_bydbql: execute a validated BydbQL statement when data retrieval is requested.

The host model performs natural language to BydbQL translation using these references and live schema discovery. Promote get_generate_bydbql_prompt to a required step whenever ranking or sorting is involved (or once schema has been fetched), because it carries the only index-rule ORDER BY check; validate_bydbql is parse-only and cannot catch a sort on a non-indexed field.

Output

For generation-only BydbQL requests, return the single BydbQL statement and state that it was not executed.

For BydbQL execution requests, show the BydbQL statement, then summarize the returned data or error concisely.

Compiling SkyWalking BanyanDB

Follow these steps to compile the project.

Step 1: Generate protobuf and mock code

Run make generate first. This regenerates .pb.go files from .proto definitions and mock files via mockgen.

This step is required whenever:

  • Proto files (.proto) have been added or modified
  • Go interfaces used by mockgen have changed

If generate fails, check:

  • protoc and Go protobuf plugins are installed
  • Proto file syntax is valid
  • Interface signatures match mock expectations

Step 2: Build all binaries

Run make build to compile all project components: ui, banyand, bydbctl, mcp, fodc/agent, fodc/proxy.

Binaries are output to each component's build/bin/dev/ directory.

Troubleshooting

  • Missing fields in .pb.go: Proto files were updated but make generate was not run. Run it first.
  • Mock generation failures: An interface changed but the mock wasn't regenerated. make generate fixes this.
  • Import errors: Check import aliases match CLAUDE.md conventions (e.g., commonv1, databasev1).

Other useful targets

  • make clean — clean all build artifacts
  • make clean-build — clean only build binaries
  • make lint — run linters
  • make test — run tests

Writing a BanyanDB Issue

BanyanDB's issues live in the apache/skywalking tracker, not in this repository — apache/skywalking-banyandb has issues disabled. Label them database, which is the component label for BanyanDB.

A good issue is one somebody can implement from without asking you a question. That is a higher bar than "clear", and it is the bar that decides whether an issue sits untouched for a year.

Size the work before polishing the issue

Issue length is not issue size. Inspect the design and the relevant production code before drafting. Count independently testable behaviour, integration seams, callers, format families, lifecycle domains, fixture work, and test suites. A short issue that says "replace the index" can be much larger than a long issue that fixes one query operation.

Always produce a short size audit during review, even if it will not appear in the filed body:

## Size audit
Classification: executable leaf | tracking parent
Boundary: <one seam, or the several seams that make this a parent>
Production activation: <the caller switched by this merge>
RED test: <the command and the observable failure on current main>
End to end: <the real operation proved by the merge>
Format/lifecycle scope: <sections or algorithms; read/write/publish/merge/GC/etc.>
Fixtures and oracle: <who supplies expected values independently of the code under test>
Focused suites: <commands>
Dependencies present on main: yes | no — <evidence>
Decision: <fits one run, or the exact split required>

Do not infer the answers only from the proposal. Use repository evidence such as interfaces, call sites, packages, fixtures, and existing tests. Treat line count as supporting evidence, never as the sole sizing rule.

Executable leaf versus tracking parent

An executable leaf must satisfy all of these:

  • It exposes one agreed boundary and delivers one behaviour end to end.
  • The same merge switches at least one named production or CLI caller to it.
  • Every requirement has a RED test that fails on current main for a stated reason and passes after the change.
  • One focused end-to-end scenario proves the real caller, not just an isolated codec or helper.
  • Expected values and persisted bytes come from an independent oracle.
  • Its prerequisites and the seam it builds on are already merged to main.
  • A contract author can create the boundary and RED tests in one workflow turn, and an implementer can reasonably complete the production change in the next.

A tracking parent describes a milestone, replacement, or multi-PR outcome. It owns completion criteria, the dependency graph, and links to children, but it is not itself an implementation contract. Mark it clearly as a tracking parent and do not put it in an automated implementation queue.

Mandatory split triggers

Classify the issue as a parent or split it before filing when any of these is true:

  • It mixes lifecycle domains such as read, write, publish/recovery, merge/GC, or replication.
  • It introduces several independent public seams or several unrelated caller cutovers.
  • It replaces a whole third-party subsystem rather than one observable operation.
  • No test can be named that fails on current main without asserting private implementation details.
  • It needs the production implementation to generate its own expected fixture bytes; that is a circular oracle, not a test contract.
  • A required boundary exists only in an open or planned PR. Keep the issue as a parent and decompose it after the blocker merges.
  • It is foundation-only: the merge adds format readers, helpers, or abstractions that no production/CLI path uses.

Use this score as a second check when none of the hard triggers is decisive:

addsignal
+2each additional public boundary after the first
+2each additional lifecycle domain after the first
+1each additional production caller family
+1each additional persisted format or algorithm family
+1each additional independent fixture family
+1each additional focused package test suite
+2a prerequisite or seam is not yet on main

Score 0–2: normally a leaf. Score 3–4: split unless repository evidence shows it is still one small behaviour. Score 5+: a tracking parent. A hard trigger overrides the score.

For a TDD workflow, also cap a leaf at roughly four observable requirements, one fixture family, one activation point, and one focused end-to-end scenario. These are review budgets, not excuses to hide work by combining requirements.

RED-test feasibility catches false slices

A compatibility-preserving cutover often returns the same result before and after the change, so a new output assertion may pass against the legacy path and is not RED. State why the test fails before implementation. Absence of a new boundary can be a compile-time RED test, but acceptance must also prove that a real caller uses the boundary; do not use brittle source-text assertions as the only proof.

If a fixture is required, include its cost in the size audit. Persisted expected bytes should be generated by an independent legacy/reference writer and checked in with provenance before the native code consumes them. Do not create a fixture-only or codec-only ticket when the project requires every merge to activate production behaviour; put the smallest fixture and decoder beside the first real caller that uses them.

The body is the specification

Assume the implementer has your issue and the codebase, and nothing else — no context from a design meeting, no follow-up in the comments. Comments are where a specification goes to be missed; if a discussion changed the plan, edit the body.

Four things have to be derivable from it:

A boundary. The seam the change exposes: a Go interface, the exported functions, the protobuf message, the HTTP surface. "Add an Authorizer that decides whether a principal may perform an operation on a group" is a boundary. "Improve permissions" is not.

Expected values from an independent source. State the concrete answers, not just the behaviour. A test whose expected value is derived the way the implementation derives it passes by construction and can never disagree with the code — so the issue is where the real answers come from. Write a reader scoped to group "ecommerce" sees only that group, not scoping should work.

Acceptance criteria provable by a command. If a criterion is "a research document" or "benchmark results", nobody can tell whether it is met. Those are worth doing and belong in a proposal, not in an implementable issue.

Where it lives and how that area is tested. Name the packages or workspace. It saves the implementer a discovery pass and it scopes the suite they run:

areatests
Go — banyand/, pkg/make test-ci PKG=./banyand/trace ./pkg/index
Canopy — canopy/npm run -w web test, npm run -w server test
Canopy end-to-endnpx playwright test --config=e2e/playwright.config.ts
everything, before pushingmake generate && make build && make pre-push

What a database issue owes that a service issue does not

Weight the criteria by what fails silently and late. If the change touches any of these, say so in the issue — the implementer should not have to infer it:

  • Anything persisted or on the wire. A segment, an index, a protobuf message: existing data must still read, and a rolling upgrade must survive a mixed-version cluster. State what an old reader does with new bytes and what a new reader does with old ones. The project labels this class bydb file compatible change.
  • Durability and ordering. For a flush, a WAL append, a compaction or a snapshot: what is the state after a crash between any two steps?
  • Concurrency. Goroutine launches go through the panic-recovery wrappers (run.Go / run.GoOrDie / run.GoWithSignal); a raw go func( needs an explicit //panicdiag:allow-rawgo directive and the lint baseline only ever shrinks.
  • Resource bounds. An unbounded buffer, a query that materialises a whole series, a retention path that cannot reclaim.

A shape that works

## Summary
One paragraph: what this delivers and why.

## Boundary
The seam this exposes — the interface, exported functions, or protobuf/HTTP
surface. What callers may rely on, and what stays free to change.

## Requirements
R1. <observable behaviour, with its concrete expected value>
R2. ...
one per behaviour, each independently checkable.

## Acceptance criteria
- every requirement has a test that fails today and passes after
- end-to-end: <a real use of the feature, start to finish>
- <compatibility / durability / concurrency criteria, where they apply>

## Scope
Packages: banyand/trace, pkg/index — or canopy/web.
Out of scope: <the neighbouring thing this is NOT>

Breaking a design-ready issue into tickets

When an issue carries a design and a proof of concept, it is usually several pieces of work. Split it into vertical slices — each cutting a complete path through the layers it touches, each verifiable on its own — and file them as sub-issues of the base.

A slice is the right size when it delivers one behaviour end to end and its acceptance criteria can be checked by running something. A slice that only makes sense alongside the next one is not a slice; it is half of one.

Do not slice a storage replacement horizontally by format section (container, postings, stored fields) if those tickets would be unused foundations. Slice by fixture complexity against one real caller instead. A first read slice may activate document count on a deliberately small fixture; a later slice can widen that same caller to multiple segments and deletion masks; another caller can then activate stored-field decoding. Each merge stays TDD-sized and live.

Decompose just in time. Keep downstream design workstreams as tracking parents until their blocker lands. Then inspect the boundary that actually merged and create their executable leaves. Pre-filing speculative leaves commits to stale seams and recreates horizontal slicing at the issue layer.

Order them by dependency and say so. Put Blocked by: #NNNN in the body of anything that needs an earlier ticket, and create them in that order so the numbers ascend with the graph. Do not start a blocked ticket before its blocker has merged — not merely opened a pull request. Until the merge, main does not contain the seam the next ticket builds on.

A pure refactor is a poor ticket if the project's convention is to land behaviour with a failing test first: a change that alters no behaviour has no test that fails today. Either land the refactor on its own as an explicit cleanup, or fold it into the first ticket that adds behaviour.

Wide, mechanical changes use expand–contract, one ticket per phase:

  1. Expand — add the new form alongside the old.
  2. Migrate — move call sites in batches sized by blast radius.
  3. Contract — delete the old form once nothing uses it.

Filing them

Create each child, then attach it to the base by the child's id, which is not its number:

url=$(gh issue create --repo apache/skywalking \
        --title "..." --body-file ./ticket-1.md --label database)
n=${url##*/}
id=$(gh api repos/apache/skywalking/issues/$n -q .id)
gh api --method POST repos/apache/skywalking/issues/<base>/sub_issues -F sub_issue_id=$id

-F, not -f: the endpoint takes an integer, and -f sends the string form. Check the result with:

gh api repos/apache/skywalking/issues/<base> -q .sub_issues_summary

The base issue stays open as the umbrella, and sub_issues_summary is the progress bar. Close it when the last ticket merges.

If automation selects the oldest issue with a queue label such as Backlog, apply that label only to the oldest unblocked executable leaf. Never queue a tracking parent or all sibling leaves at once. Create children in dependency order so issue numbers reinforce rather than fight the execution order.

Before you file

  • Did you show the size audit and explicitly classify the issue as a leaf or a tracking parent?
  • Could someone write a failing test from this body alone, without asking you anything?
  • Does the test actually fail on current main, or would the legacy path already satisfy it?
  • Does every expected value come from the issue rather than from what the code would compute?
  • Is every acceptance criterion provable by running something?
  • Is it one boundary and one production activation, or a plan with phases?
  • Does this merge activate a real caller rather than land dead foundation code?
  • Are all prerequisite seams already merged to main?
  • If it touches stored or wire data, does it say what happens to existing data and to a mixed-version cluster?

Creating a Pull Request for SkyWalking BanyanDB

Branch Rules

Always create a PR from a new branch. Never push directly to main. This is the same convention as the main Apache SkyWalking repo (https://github.com/apache/skywalking).

git checkout -b <descriptive-branch-name>

Local Checks Before Creating PR

Run these checks locally before pushing. They mirror the CI check job and PR-blocking tests in .github/workflows/ci.yml.

Required: Code Generation and Build

make generate
make build

Required: Linting and Formatting

make lint
make check

make check verifies formatting (gofumpt), go mod tidy, and ensures no uncommitted generated file diffs.

Why this order: build must succeed before lint so generated code exists. check validates consistency after linting.

After make lint passes: If lint introduced any fixes (e.g. auto-formatting, field alignment corrections), commit those changes before running make check. Do NOT update CHANGES.md for these fixup commits — just stage all modified tracked files and commit with a message like chore: fix lint issues. Then run make check on the clean tree.

Required: License Headers

make license-check

All source files must have Apache 2.0 license headers.

Required: Update CHANGES.md

Add a one-line entry under the current development version section in CHANGES.md (at the repo root). Place it under the appropriate subsection (### Features, ### Bug Fixes, etc.).

Unit Tests

Run these test packages. Each can run in parallel if the user's machine has enough cores, but it's fine to run them sequentially:

make test-ci PKG=./banyand/...
make test-ci PKG=./bydbctl/...
make test-ci PKG=./pkg/...
make test-ci PKG=./fodc/...

The CI uses these options: --vv --fail-fast --label-filter \!slow with coverage flags. For local runs, use a simplified form unless the user asks for full CI parity:

TEST_CI_OPTS="--vv --fail-fast --label-filter \!slow" make test-ci PKG=./banyand/...

Integration Tests

Run these after unit tests pass:

make test-ci PKG=./test/integration/standalone/...
make test-ci PKG=./test/integration/distributed/...

Checks NOT Practical Locally

These run in CI only — no need to run locally:

  • e2e tests — require Docker + OAP stack (90 min timeout)
  • fodc-e2e tests — require Kind Kubernetes cluster
  • dependency-review — GitHub-specific action
  • slow/flaky/property-repair tests — scheduled, not PR-blocking

Common issues

  • make lint fails with field alignment errors: The linter reports structs with suboptimal field ordering (e.g. fieldalignment: struct with X pointer bytes could be Y). Fix them automatically with:

    ~/go/bin/fieldalignment -fix ./path/to/package/...
    

    Parse the lint output to find which packages have alignment issues, run fieldalignment -fix on those packages, then re-run make lint to confirm they're resolved.

  • make lint fails with formatting errors: Run make format to auto-fix, then re-run make lint to confirm.

  • Test timeout: Integration tests can be slow; add TEST_CI_OPTS="--timeout 60m" if they time out

  • Missing tools: make check-req will tell you what's missing

  • make lint or make check fails with buf: not found: Install buf automatically by running make -C api generate, then retry.

  • make build fails: Check if generated files are up to date with make generate

Creating the PR

git push -u origin <branch-name>
gh pr create --title "<title>" --body "<body>"

Follow the standard PR format with a summary and test plan.

After the PR Is Merged

Merged branches must be removed, both locally and on origin, so they do not pile up. Do this once the PR has actually been merged — never for an open PR, and never for a PR that was closed without merging unless the user asks.

Confirm the merge state first:

gh pr view <number> --json state,mergedAt --jq '"\(.state) \(.mergedAt)"'

Only proceed when the state is MERGED. Then clean up:

git checkout main
git pull --ff-only origin main
git branch -D <branch-name>
git push origin --delete <branch-name>
git fetch --prune origin

Notes:

  • This repo only allows squash merges, so the squash commit on main never matches the branch's own commits. git branch -d therefore refuses to delete even a merged branch; -D is required, which is why the MERGED check above comes first.

  • If git push origin --delete reports remote ref does not exist, GitHub already removed the remote branch. That is fine; continue with git fetch --prune.

  • Also prune any other local branches whose upstream is already gone (earlier merged PRs). List them, confirm with the user that each was merged, then delete:

    git fetch --prune origin
    git branch -vv | grep ': gone\]' | awk '{print $1}'
    

Upgrade vendor dependencies

Run the vendor-update Make target across all projects:

make vendor-update

This runs go get -u ./... and go mod tidy -compat=1.26 in each project directory (ui, banyand, bydbctl, mcp, fodc/agent, fodc/proxy, pkg, test).

Update license dependencies

After the vendor upgrade completes, run make license-dep to update any license-related files that may have changed due to new or updated dependencies:

make license-dep

This ensures license headers and dependency license files stay in sync with the upgraded vendors.

Check for tool version drift

After upgrading, compare go.mod dependency versions against scripts/build/version.mk to find any tools whose library version was bumped and may need their tool version updated.

Version mapping

Read scripts/build/version.mk and go.mod in parallel. For each tool version variable, check if the corresponding library in go.mod was upgraded:

version.mk variableBinarygo.mod package
BUF_VERSIONbin/bufgithub.com/bufbuild/buf (if present)
PROTOC_GEN_GO_VERSIONbin/protoc-gen-gogoogle.golang.org/protobuf
PROTOC_GEN_GO_GRPC_VERSIONbin/protoc-gen-go-grpcgoogle.golang.org/grpc
PROTOC_GEN_DOC_VERSIONbin/protoc-gen-docgithub.com/pseudomuto/protoc-gen-doc (if present)
GRPC_GATEWAY_VERSIONbin/protoc-gen-grpc-gatewaygithub.com/grpc-ecosystem/grpc-gateway/v2
PROTOC_GEN_VALIDATE_VERSIONbin/protoc-gen-validategithub.com/envoyproxy/protoc-gen-validate
GOLANGCI_LINT_VERSIONbin/golangci-lint(no go.mod entry — only in version.mk)
REVIVE_VERSIONbin/revive(no go.mod entry — only in version.mk)
LICENSE_EYE_VERSIONbin/license-eye(no go.mod entry — only in version.mk)
MOCKGEN_VERSIONbin/mockgengo.uber.org/mock
GINKGO_VERSIONbin/ginkgogithub.com/onsi/ginkgo/v2
GOVULNCHECK_VERSIONbin/govulncheckgolang.org/x/vuln
BPF2GO_VERSION(used in bpf2go.mk)github.com/cilium/ebpf

Only update version.mk if the library version in go.mod has changed and the tool version should follow. Many tools (golangci-lint, revive, license-eye) are not in go.mod — their versions only change when explicitly updated in version.mk, not from make vendor-update.

Update version.mk if needed

If a tool version should be bumped (library changed and tool version needs to match), update scripts/build/version.mk using the Edit tool. Keep the format exactly — e.g., PROTOC_GEN_GO_VERSION := v1.36.11.

Remove stale binaries

When version.mk is updated, remove the corresponding old binary from bin/:

rm bin/<binary-name>

This ensures the next build/install of that tool fetches the correct new version.

MCP servers 1

Sponsors

Details

Marketplace
apache/skywalking-banyandb
Category
Community