axiomhq avatar

axiom-cli

axiomhq/cliOfficial

APL query assistance and Axiom CLI skills for Claude Code

version
0.22.0
GitHub stars
60

Install

/plugin install axiom-cli@axiom-cli

Run it inside Claude Code after adding the marketplace with /plugin marketplace add axiomhq/cli.

Contents

Discovered from source

Skills 4

Axiom Processing Language (APL)

APL is Axiom's query language for analyzing observability data. This skill provides comprehensive guidance for writing, debugging, and optimizing APL queries.

Quick Reference

Documentation: https://axiom.co/docs/apl/introduction

CLI usage: See references/cli.md

Core Workflow

1. List Available Datasets

axiom dataset list -f json

2. Discover Schema (CRITICAL - Always Do First)

['<dataset>'] | getschema

Never guess field names. The schema shows all fields with their types.

3. Sample Data

['<dataset>'] | limit 10

4. Write Query

See references for operators, functions, and patterns.

APL Syntax Essentials

Dataset Reference

['dataset-name']           // Bracket notation (required for names with dots/dashes)
dataset_name               // Plain identifier (only for simple names)

Field Reference

field_name                 // Plain field
['field.with.dots']        // Bracket notation for dotted fields
['service.name']           // OTel data (see references/otel.md for field mappings)

Basic Query Structure

['dataset']
| where <condition>
| extend <new_field> = <expression>
| summarize <aggregation> by <grouping>
| project <fields>
| sort by <field> desc
| limit 100

Time Handling

Always filter by time first - it's the most selective filter.

// Relative time
| where _time >= ago(1h)
| where _time >= ago(24h) and _time < ago(1h)

// Absolute time
| where _time >= datetime(2024-01-15T10:00:00Z)
| where _time between (datetime(2024-01-15) .. datetime(2024-01-16))

Time functions:

  • ago(timespan) - Relative past time
  • now() - Current time
  • datetime(string) - Parse datetime
  • bin(_time, 5m) - Time bucketing
  • bin_auto(_time) - Automatic bucketing

When NOT to Use

  • Simple field lookup: Use getschema directly instead of invoking the full skill
  • Known query patterns: If you already have a working query, don't re-invoke for syntax help
  • Real-time alerting: Use Axiom Monitors for continuous alerting, not ad-hoc queries

References

Anomaly Detection

Detect anomalies in Axiom datasets by comparing recent patterns to historical baselines using statistical analysis.

Arguments

When invoked with a dataset name (e.g., /detect-anomalies logs), it's available as $ARGUMENTS.

Prerequisites

Statistical anomaly detection requires sufficient data:

  • Minimum data points: Z-score and standard deviation need ≥30 samples per bucket for statistical significance
  • Historical baseline: At least 24 hours of data for meaningful comparison (methods use 25h lookback)
  • Consistent ingestion: Gaps in data collection will skew baselines

If these aren't met, results may be misleading. Consider using simpler threshold-based alerting instead.

Schema Discovery

Always verify field names first:

axiom query "['<dataset>'] | getschema" --start-time -1h

Anomaly Detection Methods

1. Volume Anomaly Detection

Compare recent volume to baseline:

Calculate baseline (past 24h excluding last hour):

axiom query "['<dataset>']
| where _time between (ago(25h) .. ago(1h))
| summarize count() by bin(_time, 1h)
| summarize
    avg_hourly = avg(count_),
    stdev_hourly = stdev(count_)" --start-time -25h -f json

Check recent volume:

axiom query "['<dataset>']
| where _time >= ago(1h)
| summarize
    current_count = count(),
    current_hour = min(_time)" --start-time -1h -f json

Z-score calculation:

  • z_score = (current - avg) / stdev
  • |z_score| > 2 indicates anomaly

2. New Value Detection

Find values that appeared recently but weren't seen before:

axiom query "['<dataset>']
| where _time >= ago(25h)
| summarize first_seen = min(_time) by error_code
| where first_seen >= ago(1h)" --start-time -25h -f json

Replace error_code with any categorical field (service, endpoint, status).

3. Statistical Outliers

Find values outside normal distribution:

Calculate bounds:

axiom query "['<dataset>']
| where _time between (ago(25h) .. ago(1h))
| summarize
    avg_val = avg(duration),
    stdev_val = stdev(duration)
| extend
    lower_bound = avg_val - 3 * stdev_val,
    upper_bound = avg_val + 3 * stdev_val" --start-time -25h -f json

Find outliers:

axiom query "['<dataset>']
| where _time >= ago(1h)
| where duration < <lower_bound> or duration > <upper_bound>
| limit 100" --start-time -1h -f json

4. Rare Event Detection

Find infrequent occurrences:

axiom query "['<dataset>']
| where _time >= ago(1h)
| summarize count() by error_message
| where count_ == 1" --start-time -1h -f json

5. Error Rate Spike

Compare error rate to baseline:

axiom query "['<dataset>']
| where _time >= ago(6h)
| summarize
    total = count(),
    errors = countif(status >= 500)
  by bin(_time, 15m)
| extend error_rate = errors * 100.0 / total
| sort by _time asc" --start-time -6h -f json

6. Latency Degradation

Track percentile changes:

axiom query "['<dataset>']
| where _time >= ago(6h)
| summarize
    p50 = percentile(duration, 50),
    p95 = percentile(duration, 95),
    p99 = percentile(duration, 99)
  by bin(_time, 15m)
| sort by _time asc" --start-time -6h -f json

Anomaly Categories

TypeDetection MethodIndicates
Volume SpikeZ-score on countTraffic surge, attack, incident
Volume DropZ-score on countOutage, data collection issue
New ValuesFirst-seen time (min(_time))New errors, new services
Statistical Outlier3-sigma ruleExtreme performance issue
Rare EventsCount = 1Unusual conditions
Error SpikeError rate increaseService degradation
Latency SpikePercentile increasePerformance issue

Output Format

## Anomaly Report: <dataset>

### Summary
- Analysis period: <timeframe>
- Anomalies found: <count>

### Volume Anomalies
| Time | Count | Expected | Z-Score |
|------|-------|----------|---------|
| ... | ... | ... | ... |

### New Values
- Field: `error_code`
- New values: `TIMEOUT_ERROR`, `CONNECTION_REFUSED`

### Statistical Outliers
- Field: `duration`
- Outliers: <count> events above <threshold>

### Error Rate
- Baseline: X%
- Current: Y%
- Change: +Z%

### Recommendations
1. <Investigation action>
2. <Monitoring suggestion>

Investigation Priority

  1. Assess impact - Is this affecting users?
  2. Correlate timing - What changed when anomaly started?
  3. Check related systems - Shared dependencies?
  4. Verify data quality - Is it a real issue or data problem?

When NOT to Use

  • Insufficient data: Z-score needs ≥30 data points; new datasets lack meaningful baselines
  • Known thresholds: If you have specific SLOs (e.g., "p99 < 500ms"), use direct threshold queries
  • Real-time alerting: Use Axiom Monitors for continuous anomaly detection, not ad-hoc analysis
  • Single data point: Anomaly detection compares against distributions, not individual values

APL Reference

For query syntax, invoke the axiom-apl skill which provides anomaly detection patterns and function documentation.

Dataset Exploration

Systematically explore an Axiom dataset to understand its structure, content, and potential use cases.

Arguments

When invoked with a dataset name (e.g., /explore-dataset logs), the name is available as $ARGUMENTS.

Exploration Protocol

1. List Available Datasets

If no dataset specified, list what's available:

axiom dataset list -f json

2. Schema Discovery

Always start here. Discover actual field names and types:

axiom query "['<dataset>'] | getschema" --start-time -1h

Identify:

  • Field names and types
  • Dotted fields requiring bracket notation
  • Timestamp fields
  • Key dimensions (service, status, level)

OTel trace data: If schema contains trace_id, span_id, attributes.*, note that:

  • Service fields are promoted: use ['service.name'] not ['resource.service.name']
  • Custom attributes: ['attributes.custom']['field'] with tostring() for aggregations
  • See axiom-apl skill's OTel reference for field mappings

3. Sample Data

Examine actual values:

axiom query "['<dataset>'] | limit 10" --start-time -1h -f json

Look for:

  • Data structure and relationships
  • Field value formats
  • Data quality issues

4. Volume Analysis

Understand data volume patterns:

axiom query "['<dataset>'] | summarize count() by bin(_time, 1h) | sort by _time asc" --start-time -24h

Analyze:

  • Event volume over time
  • Data freshness
  • Collection gaps

5. Categorical Field Analysis

For each key categorical field (status, level, service):

axiom query "['<dataset>'] | summarize count() by <field> | top 20 by count_" --start-time -1h

Identify:

  • Value distributions
  • Cardinality
  • Key dimensions for filtering

6. Numerical Field Statistics

For numeric fields (duration, bytes, count):

axiom query "['<dataset>'] | summarize count(), min(<field>), max(<field>), avg(<field>), percentiles(<field>, 50, 95, 99)" --start-time -1h

7. Error Pattern Detection

Search for error indicators:

axiom query "search in (['<dataset>']) 'error' or 'fail' or 'exception' | limit 20" --start-time -1h

Output Format

Provide a summary including:

## Dataset Summary: <name>

### Purpose
<What system generated this data, what it represents>

### Key Fields
| Field | Type | Description |
|-------|------|-------------|
| ... | ... | ... |

### Volume
- Events per hour: ~X
- Data freshness: last event at X

### Key Dimensions
- `status`: 200, 400, 500, ...
- `service.name`: api, web, worker, ...

### Recommended Queries
<Common queries for this dataset>

### Monitoring Opportunities
<What could be alerted on>

When NOT to Use

  • Known datasets: If you already understand the schema, skip exploration and query directly
  • Quick field check: Use getschema directly for single field lookups
  • Production queries: Exploration uses expensive operations (search); extract patterns then optimize
  • Repeated analysis: Once explored, document findings and reuse—don't re-explore

APL Reference

For query syntax, invoke the axiom-apl skill which provides comprehensive documentation on operators, functions, and patterns.

Trace Analysis

Analyze OpenTelemetry distributed traces to identify errors, latency issues, and root causes.

Arguments

When invoked with a trace ID (e.g., /find-traces abc123...), it's available as $ARGUMENTS.

Trace Dataset Discovery

First, find trace datasets:

axiom dataset list -f json

Look for datasets containing trace data (often named *traces*, *spans*, or otel-*).

Schema Discovery

Always verify field names first:

axiom query "['<trace-dataset>'] | getschema" --start-time -1h

Common Operations

Get Trace by ID

axiom query "['<dataset>']
| where trace_id == '<TRACE_ID>'
| sort by _time asc
| limit 100" --start-time -1h -f json

Find Error Traces

axiom query "['<dataset>']
| where _time >= ago(1h)
| where error == true
| extend error = coalesce(ensure_field(\"error\", typeof(bool)), false)
| summarize
    start_time = min(_time),
    total_duration = max(duration),
    span_count = count(),
    error_count = countif(error),
    services = make_set(['service.name']),
    arg_min(_time, root_operation = name)
  by trace_id
| sort by start_time desc
| limit 20" --start-time -1h -f json

Find Slow Traces

axiom query "['<dataset>']
| where _time >= ago(1h)
| where duration >= 1000000000
| summarize
    start_time = min(_time),
    total_duration = max(duration),
    span_count = count(),
    services = make_set(['service.name'])
  by trace_id
| sort by total_duration desc
| limit 20" --start-time -1h -f json

Find Traces by Service

axiom query "['<dataset>']
| where _time >= ago(1h)
| where ['service.name'] == '<SERVICE>'
| summarize
    start_time = min(_time),
    total_duration = max(duration),
    span_count = count(),
    error_count = countif(error == true)
  by trace_id
| sort by start_time desc
| limit 20" --start-time -1h -f json

Error Spans in Trace

axiom query "['<dataset>']
| where trace_id == '<TRACE_ID>'
| where error == true
| project _time, ['service.name'], name, duration, ['status.message']" --start-time -1h -f json

Critical Path Analysis

axiom query "['<dataset>']
| where trace_id == '<TRACE_ID>'
| project span_id, parent_span_id, ['service.name'], name, duration, error
| sort by duration desc" --start-time -1h -f json

OTel Field Reference

FieldBracket?Description
trace_idNo32-char trace identifier
span_idNo16-char span identifier
parent_span_idNoParent span (empty for root)
nameNoOperation name
durationNoDuration in nanoseconds
kindNoCLIENT, SERVER, INTERNAL, PRODUCER, CONSUMER
errorNoBoolean error flag
['service.name']YesService identifier
['status.code']YesOK, ERROR, or nil
['status.message']YesError description
['scope.name']YesInstrumentation library

Duration Conversion

OTel durations are in nanoseconds:

HumanNanosecondsFilter
1 ms1,000,000duration >= 1000000
100 ms100,000,000duration >= 100000000
1 s1,000,000,000duration >= 1000000000

Convert for display:

| extend duration_ms = duration / 1000000.0

Custom Attributes

Non-standard span attributes are stored in attributes.custom map:

// Filter by custom attribute
| where ['attributes.custom']['user_id'] == "123"

// Aggregation requires explicit cast
| summarize count() by tostring(['attributes.custom']['tenant'])

Without tostring(), aggregations fail with "grouping by field of type unknown".

Codebase Correlation

When working in a repository that matches the traced service, correlate trace data with source code to identify root causes.

Mapping Trace Data to Code

  1. Extract package/module path from ['scope.name']

    • Contains the instrumentation library or package path
    • Strip the module prefix to get the local path
    • Example: github.com/org/repo/pkg/auth → pkg/auth
  2. Find code from operation name

    • The name field often contains function names or HTTP routes
    • Search the codebase for matching handlers, functions, or endpoints
  3. Trace the call chain

    • Follow parent-child span relationships
    • Map each span to its corresponding code location
    • Identify where errors originate and propagate

Note: Codebase correlation is optional. Proceed with trace-only analysis if code is unavailable or doesn't match the traced services.

Output Format

When analyzing a trace, provide:

## Trace Summary
- **Trace ID:** <id>
- **Duration:** <human-readable>
- **Services:** <list>
- **Outcome:** success/failure

## Sequence of Events
1. <Service> - <operation> (<duration>)
2. <Service> - <operation> (<duration>) ⚠️ ERROR
...

## Error Analysis
<What failed, when, why>

## Root Cause
<Deepest error and explanation>

## Codebase Locations (if applicable)
- **Service:** <service.name>
- **Package:** <scope.name>
- **Files:** <specific files to investigate>

## Recommended Actions
1. <Specific action>
2. <What to investigate next>

When NOT to Use

  • Metrics analysis: Traces are for request flow; use logs/metrics skills for aggregated performance data
  • Non-OTel data: This skill assumes OpenTelemetry field conventions (trace_id, span_id, etc.)
  • Known trace structure: If you already have the query, run it directly without invoking this skill
  • Alerting on trace patterns: Use Axiom Monitors for continuous alerting

APL Reference

For query syntax, invoke the axiom-apl skill which provides trace analysis patterns and duration unit guidance.

Sponsors

Details

Marketplace
axiomhq/cli
Author
Axiom, Inc.
Category
Community
License
MIT
Keywords
axiom, observability, apl, logs, traces, metrics