marslanalvi avatar

data-data-engineering-analytics

Data pipelines (Airflow, dbt, Spark), analytics, dashboards and metrics

version
1.0.0
GitHub stars
5

Install

/plugin install data-data-engineering-analytics@skills-stack

Run it inside Claude Code after adding the marketplace with /plugin marketplace add marslanalvi/SkillsStack.

Contents

Discovered from source

Skills 9

Apache Airflow DAG Patterns

Production-ready patterns for Apache Airflow including DAG design, operators, sensors, testing, and deployment strategies.

When to Use This Skill

  • Creating data pipeline orchestration with Airflow
  • Designing DAG structures and dependencies
  • Implementing custom operators and sensors
  • Testing Airflow DAGs locally
  • Setting up Airflow in production
  • Debugging failed DAG runs

Core Concepts

1. DAG Design Principles

PrincipleDescription
IdempotentRunning twice produces same result
AtomicTasks succeed or fail completely
IncrementalProcess only new/changed data
ObservableLogs, metrics, alerts at every step

2. Task Dependencies

# Linear
task1 >> task2 >> task3

# Fan-out
task1 >> [task2, task3, task4]

# Fan-in
[task1, task2, task3] >> task4

# Complex
task1 >> task2 >> task4
task1 >> task3 >> task4

Quick Start

# dags/example_dag.py
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.operators.empty import EmptyOperator

default_args = {
    'owner': 'data-team',
    'depends_on_past': False,
    'email_on_failure': True,
    'email_on_retry': False,
    'retries': 3,
    'retry_delay': timedelta(minutes=5),
    'retry_exponential_backoff': True,
    'max_retry_delay': timedelta(hours=1),
}

with DAG(
    dag_id='example_etl',
    default_args=default_args,
    description='Example ETL pipeline',
    schedule='0 6 * * *',  # Daily at 6 AM
    start_date=datetime(2024, 1, 1),
    catchup=False,
    tags=['etl', 'example'],
    max_active_runs=1,
) as dag:

    start = EmptyOperator(task_id='start')

    def extract_data(**context):
        execution_date = context['ds']
        # Extract logic here
        return {'records': 1000}

    extract = PythonOperator(
        task_id='extract',
        python_callable=extract_data,
    )

    end = EmptyOperator(task_id='end')

    start >> extract >> end

Detailed patterns and worked examples

Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.

Best Practices

Do's

  • Use TaskFlow API - Cleaner code, automatic XCom
  • Set timeouts - Prevent zombie tasks
  • Use mode='reschedule' - For sensors, free up workers
  • Test DAGs - Unit tests and integration tests
  • Idempotent tasks - Safe to retry

Don'ts

  • Don't use depends_on_past=True - Creates bottlenecks
  • Don't hardcode dates - Use {{ ds }} macros
  • Don't use global state - Tasks should be stateless
  • Don't skip catchup blindly - Understand implications
  • Don't put heavy logic in DAG file - Import from modules

Calibrate First

Overview

A backtest result is only as good as the instrument that produced it. This skill makes Claude measure the instrument before it measures the signal. It drives assay-lite, a single-file, offline Python tool. floor injects synthetic signals of known strength into the user's own returns to find the smallest effect the data can detect. test charges every run to an append-only registry before computing, and a result below the floor is reported as UNDECIDABLE rather than "the signal doesn't work".

When to Use

  • The user asks whether a trading signal, factor or ranking "works", is "significant", or has an "edge".
  • The user wants to read an information coefficient (IC) or a cross-sectional backtest result, especially a null one.
  • The user is about to try several variants of an idea and pick the best one.

Do not use it for single-asset timing signals, intraday data, live trading or position sizing. assay-lite does not cover them.

How It Works

Use only a previously reviewed local assay_lite.py at an explicit absolute path, verified against commit 78104a4356541affe2a469dfdf69cb1e418eb0c5, together with a fully locked and reviewed numpy/pandas/scipy environment. This skill does not download code or resolve dependencies. If absent, stop before execution and return the input schema and analysis plan. Do not substitute a same-named project file. A separate installation task must stage and inspect every file and dependency with installation hooks disabled, verify immutable bytes, then obtain approval before executing or writing the trial registry. uv caching is not isolation.

Four rules:

  1. Run floor before interpreting any result. Until the floor is measured for these prices and this horizon, a non-significant result means nothing either way.
  2. Never compare p with a conventional 0.05. The only bars are the ones the registry prints: 0.05 divided by every trial ever charged. "Bar for the last trial charged" and "the NEXT test will face" are different bars. Quote the one that matches the question.
  3. Trials are never refunded. Re-running a signal, tweaking a parameter or trying another horizon is a new trial. Never suggest deleting registry lines or using a fresh --home to reset the count.
  4. Below the floor is UNDECIDABLE, not "no effect". Say "too small for this data to see", never "doesn't work".

Procedure:

  1. Check the inputs. Prices are date,asset,close. The signal is date,asset,signal, known at that date's close.
  2. Run breadth, and state the effective number of independent bets next to the asset count.
  3. Run floor for the exact prices file and horizon you will test.
  4. Write down in one sentence what you expect and why, then run test once.
  5. Report the verdict line verbatim, with IC, p, the bar it faced, the dispersion ratio and the scaled floor.

Examples

python /absolute/reviewed/assay_lite.py demo          # synthetic data; shows PASS, UNDECIDABLE, and a scaled floor
python /absolute/reviewed/assay_lite.py breadth  --prices prices.csv
python /absolute/reviewed/assay_lite.py floor    --prices prices.csv --horizon 1
python /absolute/reviewed/assay_lite.py test     --prices prices.csv --signal signal.csv --name momentum_12_1 --horizon 1
python /absolute/reviewed/assay_lite.py registry
verdictsay thisnot this
PASS"cleared the multiple-testing bar""proven", "profitable"
UNDECIDABLE"too small for this data to see""doesn't work"
FAIL"large but unreliable""zero"

Limitations

  • Cross-sectional and daily only, using Spearman IC. No timing signals and no portfolio simulation.
  • No transaction costs. A PASS says nothing about profit after costs.
  • Bonferroni is conservative when trials are correlated.
  • The floor is calibrated with independent noise and then scaled up by each signal's daily-IC dispersion. That corrects for factor structure on average, not exactly.
  • Nothing here is investment advice. A PASS is evidence of a ranking effect, not permission to trade.

Security & Safety Notes

  • assay-lite makes no network calls and needs no API keys. It reads the CSVs the user points it at and writes only to its own .assay-lite/ folder.
  • The registry is append-only on purpose. Don't edit or delete it on the user's behalf.

Data Quality Frameworks

Production patterns for implementing data quality with Great Expectations, dbt tests, and data contracts to ensure reliable data pipelines.

When to Use This Skill

  • Implementing data quality checks in pipelines
  • Setting up Great Expectations validation
  • Building comprehensive dbt test suites
  • Establishing data contracts between teams
  • Monitoring data quality metrics
  • Automating data validation in CI/CD

Core Concepts

1. Data Quality Dimensions

DimensionDescriptionExample Check
CompletenessNo missing valuesexpect_column_values_to_not_be_null
UniquenessNo duplicatesexpect_column_values_to_be_unique
ValidityValues in expected rangeexpect_column_values_to_be_in_set
AccuracyData matches realityCross-reference validation
ConsistencyNo contradictionsexpect_column_pair_values_A_to_be_greater_than_B
TimelinessData is recentexpect_column_max_to_be_between

2. Testing Pyramid for Data

          /\
         /  \     Integration Tests (cross-table)
        /────\
       /      \   Unit Tests (single column)
      /────────\
     /          \ Schema Tests (structure)
    /────────────\

Quick Start

Great Expectations Setup

# Install
pip install great_expectations

# Initialize project
great_expectations init

# Create datasource
great_expectations datasource new
# great_expectations/checkpoints/daily_validation.yml
import great_expectations as gx

# Create context
context = gx.get_context()

# Create expectation suite
suite = context.add_expectation_suite("orders_suite")

# Add expectations
suite.add_expectation(
    gx.expectations.ExpectColumnValuesToNotBeNull(column="order_id")
)
suite.add_expectation(
    gx.expectations.ExpectColumnValuesToBeUnique(column="order_id")
)

# Validate
results = context.run_checkpoint(checkpoint_name="daily_orders")

Detailed patterns and worked examples

Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.

Summary: {total_passed}/{total_tables} tables passed")

    report.append("")

    for table, result in results.items():
        status = "✅" if result.passed else "❌"
        report.append(f"### {status} {table}")
        report.append(f"- Expectations: {result.total_expectations}")
        report.append(f"- Failed: {result.failed_expectations}")

        if not result.passed:
            report.append("- Failed checks:")
            for detail in result.details:
                if not detail["success"]:
                    report.append(f"  - {detail['expectation']}: {detail['observed_value']}")
        report.append("")

    return "\n".join(report)

Usage

context = gx.get_context() pipeline = DataQualityPipeline(context)

tables_to_validate = { "orders": "orders_suite", "customers": "customers_suite", "products": "products_suite", }

results = pipeline.run_all(tables_to_validate) report = pipeline.generate_report(results)

Fail pipeline if any table failed

if not all(r.passed for r in results.values()): print(report) raise ValueError("Data quality checks failed!")


## Best Practices

### Do's

- **Test early** - Validate source data before transformations
- **Test incrementally** - Add tests as you find issues
- **Document expectations** - Clear descriptions for each test
- **Alert on failures** - Integrate with monitoring
- **Version contracts** - Track schema changes

### Don'ts

- **Don't test everything** - Focus on critical columns
- **Don't ignore warnings** - They often precede failures
- **Don't skip freshness** - Stale data is bad data
- **Don't hardcode thresholds** - Use dynamic baselines
- **Don't test in isolation** - Test relationships too

Data Storytelling

Transform raw data into compelling narratives that drive decisions and inspire action.

When to Use This Skill

  • Presenting analytics to executives
  • Creating quarterly business reviews
  • Building investor presentations
  • Writing data-driven reports
  • Communicating insights to non-technical audiences
  • Making recommendations based on data

Core Concepts

1. Story Structure

Setup → Conflict → Resolution

Setup: Context and baseline
Conflict: The problem or opportunity
Resolution: Insights and recommendations

2. Narrative Arc

1. Hook: Grab attention with surprising insight
2. Context: Establish the baseline
3. Rising Action: Build through data points
4. Climax: The key insight
5. Resolution: Recommendations
6. Call to Action: Next steps

3. Three Pillars

PillarPurposeComponents
DataEvidenceNumbers, trends, comparisons
NarrativeMeaningContext, causation, implications
VisualsClarityCharts, diagrams, highlights

Detailed patterns and worked examples

Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.

Best Practices

Do's

  • Start with the "so what" - Lead with insight
  • Use the rule of three - Three points, three comparisons
  • Show, don't tell - Let data speak
  • Make it personal - Connect to audience goals
  • End with action - Clear next steps

Don'ts

  • Don't data dump - Curate ruthlessly
  • Don't bury the insight - Front-load key findings
  • Don't use jargon - Match audience vocabulary
  • Don't show methodology first - Context, then method
  • Don't forget the narrative - Numbers need meaning

Data Throughput Accelerator

Use this skill when the bottleneck is moving, transforming, or saving lots of data. The goal is not just speed. The goal is faster correct data landing in the right place with proof.

First Distinction

Separate these before optimizing:

  • source extraction speed;
  • network transfer speed;
  • warehouse/load speed;
  • transform speed;
  • serving-table freshness;
  • live tail growth while the job runs.

A pipeline can be "fast" and still appear behind if new data arrives faster than the final catch-up window.

Fast Path Heuristics

  • Move compute to where the data already is.
  • Prefer warehouse-native scans, joins, and appends for large landed files.
  • Use manifests or checkpoints so completed files/partitions are skipped.
  • Use partitioning and clustering that match the read and append pattern.
  • Batch small files, requests, and writes.
  • Make writes idempotent through unique keys, manifests, or replaceable staging.
  • Keep raw, derived, and serving tables separately accountable.

Workflow

  1. Read the current source, target, and manifest contracts.
  2. Measure backlog: external files, manifest rows, raw rows, derived rows, min/max timestamps, and unprocessed counts.
  3. Run a safe catch-up or sample benchmark.
  4. Compare variants: batch size, worker count, warehouse SQL, file grouping, staging shape, and manifest update method.
  5. Promote only the fastest path that keeps counts and timestamps coherent.
  6. Codify the path as a CLI, scheduled job, workflow, or runbook.
  7. Rerun final accounting after the codified path executes.

Accounting Output

Use a hard accounting block:

Data throughput result:
- Source files discovered: 294
- Files processed this run: 294
- Raw rows added: 9,683,598
- Derived rows added: 8,917,585
- Remaining tail: 24 files at readback time
- Runtime: 38.7s
- Correctness gate: manifest counts and table max timestamps match

Guardrails

  • Do not delete raw data to make a metric look better.
  • Do not skip failed files silently.
  • Do not mix historical backfill status with live-tail freshness.
  • Do not call a pipeline complete until the target tables and manifest agree.
  • For finance, healthcare, regulated, or customer-impacting data, preserve replay evidence and approval gates.

dbt Transformation Patterns

Production-ready patterns for dbt (data build tool) including model organization, testing strategies, documentation, and incremental processing.

When to Use This Skill

  • Building data transformation pipelines with dbt
  • Organizing models into staging, intermediate, and marts layers
  • Implementing data quality tests
  • Creating incremental models for large datasets
  • Documenting data models and lineage
  • Setting up dbt project structure

Core Concepts

1. Model Layers (Medallion Architecture)

sources/          Raw data definitions
    ↓
staging/          1:1 with source, light cleaning
    ↓
intermediate/     Business logic, joins, aggregations
    ↓
marts/            Final analytics tables

2. Naming Conventions

LayerPrefixExample
Stagingstg_stg_stripe__payments
Intermediateint_int_payments_pivoted
Martsdim_, fct_dim_customers, fct_orders

Quick Start

# dbt_project.yml
name: "analytics"
version: "1.0.0"
profile: "analytics"

model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]

vars:
  start_date: "2020-01-01"

models:
  analytics:
    staging:
      +materialized: view
      +schema: staging
    intermediate:
      +materialized: ephemeral
    marts:
      +materialized: table
      +schema: analytics
# Project structure
models/
├── staging/
│   ├── stripe/
│   │   ├── _stripe__sources.yml
│   │   ├── _stripe__models.yml
│   │   ├── stg_stripe__customers.sql
│   │   └── stg_stripe__payments.sql
│   └── shopify/
│       ├── _shopify__sources.yml
│       └── stg_shopify__orders.sql
├── intermediate/
│   └── finance/
│       └── int_payments_pivoted.sql
└── marts/
    ├── core/
    │   ├── _core__models.yml
    │   ├── dim_customers.sql
    │   └── fct_orders.sql
    └── finance/
        └── fct_revenue.sql

Detailed patterns and worked examples

Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.

Best Practices

Do's

  • Use staging layer - Clean data once, use everywhere
  • Test aggressively - Not null, unique, relationships
  • Document everything - Column descriptions, model descriptions
  • Use incremental - For tables > 1M rows
  • Version control - dbt project in Git

Don'ts

  • Don't skip staging - Raw → mart is tech debt
  • Don't hardcode dates - Use {{ var('start_date') }}
  • Don't repeat logic - Extract to macros
  • Don't test in prod - Use dev target
  • Don't ignore freshness - Monitor source data

KPI Dashboard Design

Comprehensive patterns for designing effective Key Performance Indicator (KPI) dashboards that drive business decisions.

When to Use This Skill

  • Designing executive dashboards
  • Selecting meaningful KPIs
  • Building real-time monitoring displays
  • Creating department-specific metrics views
  • Improving existing dashboard layouts
  • Establishing metric governance

Core Concepts

1. KPI Framework

LevelFocusUpdate FrequencyAudience
StrategicLong-term goalsMonthly/QuarterlyExecutives
TacticalDepartment goalsWeekly/MonthlyManagers
OperationalDay-to-dayReal-time/DailyTeams

2. SMART KPIs

Specific: Clear definition
Measurable: Quantifiable
Achievable: Realistic targets
Relevant: Aligned to goals
Time-bound: Defined period

3. Dashboard Hierarchy

├── Executive Summary (1 page)
│   ├── 4-6 headline KPIs
│   ├── Trend indicators
│   └── Key alerts
├── Department Views
│   ├── Sales Dashboard
│   ├── Marketing Dashboard
│   ├── Operations Dashboard
│   └── Finance Dashboard
└── Detailed Drilldowns
    ├── Individual metrics
    └── Root cause analysis

Detailed worked examples and patterns

Detailed sections (starting with ## Common KPIs by Department) live in references/details.md. Read that file when the navigation summary above is insufficient.

Best Practices

Do's

  • Limit to 5-7 KPIs - Focus on what matters
  • Show context - Comparisons, trends, targets
  • Use consistent colors - Red=bad, green=good
  • Enable drilldown - From summary to detail
  • Update appropriately - Match metric frequency

Don'ts

  • Don't show vanity metrics - Focus on actionable data
  • Don't overcrowd - White space aids comprehension
  • Don't use 3D charts - They distort perception
  • Don't hide methodology - Document calculations
  • Don't ignore mobile - Ensure responsive design

Troubleshooting

MRR shown on dashboard contradicts finance's number

The most common cause is inconsistent treatment of annual plans. Finance may prorate to a daily rate while the dashboard normalizes to monthly. Align on a single formula and document it directly on the dashboard card:

-- Explicit formula shown in tooltip / data dictionary
-- Annual plans: divide total contract value by 12
-- Quarterly plans: divide by 3
-- Monthly plans: use as-is
CASE subscription_interval
    WHEN 'monthly'   THEN amount
    WHEN 'quarterly' THEN amount / 3.0
    WHEN 'yearly'    THEN amount / 12.0
END AS normalized_mrr

Dashboard shows green but product team reports users complaining

The dashboard likely tracks system uptime (a lagging indicator) but not user-facing quality metrics. Add customer-perceived metrics alongside infrastructure metrics:

Infrastructure (green)User-perceived (add these)
API uptime 99.9%P95 page load time
Error rate 0.1%Task completion rate
Queue depth normalSupport ticket volume

Retention cohort looks flat — no variation between cohorts

Check whether the cohort query is partitioning by signup month correctly. A common bug is using created_at::date instead of DATE_TRUNC('month', created_at), which groups by day and produces cohorts too small to show trends:

-- Wrong: too granular, cohorts are too small
DATE_TRUNC('day', created_at) AS cohort_date

-- Correct: monthly cohorts
DATE_TRUNC('month', created_at) AS cohort_month

Real-time dashboard hammers the database

A live dashboard refreshing every 10 seconds with complex cohort SQL will degrade production query performance. Separate OLAP workloads from OLTP by writing pre-aggregated metrics to a summary table via a scheduled job, and have the dashboard read from that:

# Scheduled every 5 minutes via cron/Celery
def refresh_mrr_summary():
    conn.execute("""
        INSERT INTO kpi_snapshot (metric, value, snapshot_at)
        SELECT 'mrr', SUM(...), NOW()
        FROM subscriptions WHERE status = 'active'
        ON CONFLICT (metric) DO UPDATE SET value = EXCLUDED.value
    """)

Alert thresholds fire constantly, team ignores them

Static thresholds set once and never reviewed cause alert fatigue. Use dynamic thresholds based on rolling averages so alerts fire only when the metric deviates significantly from its own baseline:

# Alert if current value is > 2 standard deviations from 30-day rolling mean
def is_anomalous(current: float, history: list[float]) -> bool:
    mean = statistics.mean(history)
    stdev = statistics.stdev(history)
    return abs(current - mean) > 2 * stdev

Related Skills

  • data-storytelling - Turn dashboard findings into narratives that drive executive decisions

When to Use

  • Use when this upstream workflow matches the user's stated goal.
  • Use when the task requires the procedures documented in this skill.

Monte Carlo Context Detection

This skill determines which Monte Carlo skill or workflow best fits the user's current context. It activates reactively for ambiguous or multi-step data-related messages, gathers signals, and routes to the right skill or workflow.

Monte Carlo tool routing (required): Always call Monte Carlo MCP tools through this plugin's upstream bundled server when installed and verified for the intended account; its historical tool names are mcp__plugin_mc-agent-toolkit_monte-carlo-mcp__<tool> (e.g. mcp__plugin_mc-agent-toolkit_monte-carlo-mcp__get_alerts). Bare tool names used in this skill (get_alerts, search, get_table, …) refer to that bundled server. If the session also has a separately-configured monte-carlo-mcp server, do not route to it — it may point at a different endpoint or credentials.

This catalog tree does not bundle that server, routing CLAUDE.md, hooks, or companion skills. Verify availability and account identity; do not invent tools or silently use a different tenant. Routing does not authorize monitor creation, deployment, or other writes beyond the user's request.

Reference file for signal definitions: references/signal-definitions.md (relative to this file). Read it before routing.

When to activate this skill

This skill is activated by the CLAUDE.md routing table when:

  • The user's message relates to data quality, alerts, incidents, coverage, or Monte Carlo — but doesn't clearly match a single skill in the routing table
  • The user's intent is ambiguous or could span multiple skills
  • The user asks a broad question like "help me with my data" or "what's going on?"

When NOT to activate this skill

  • A skill or workflow is already active in the conversation — the active skill owns the conversation, do not intercept
  • The user's message clearly matches a single skill in the CLAUDE.md routing table — route directly, no need for context detection
  • The user is editing a dbt model — defer to the prevent skill only when its hooks are verified active
  • The user's message is not data-related at all

Workflow: Reactive Routing

This skill is purely reactive — it activates for ambiguous or multi-step data-related messages and routes them.

Follow these steps in order.

Step 0: Fast-path clear intent (stop early if matched)

Before doing anything else, check whether the user's message unambiguously matches a single existing skill. If so, skip the rest of this workflow and immediately load that skill — do NOT read references/signal-definitions.md, do NOT make API probes.

Clear user intentSkill to load immediately
"Check health of [named table]" / "status of [named table]"../asset-health/SKILL.md
"Create a [monitor type] on [named table]"../monitoring-advisor/SKILL.md
"Investigate alert on [named table]" / "why is [named table] stale/broken?"../incident-response/SKILL.md
"What should I monitor?" / "where are my coverage gaps?"../proactive-monitoring/SKILL.md
"Instrument my agent" / "set up Monte Carlo tracing on [named framework] agent" / "setting up an agent"../instrument-agent/SKILL.md

Context-detection is for ambiguous requests only. If the request is clear, routing through this skill wastes turns and tokens.

If no clear match, proceed to Step 1.

Step 1: Categorize intent

Read references/signal-definitions.md for the full signal catalog. Determine which category the user's message falls into:

CategorySignalsExample messages
Specific assetUser mentions a table name, or has a .sql model file open in their IDE"what's wrong with stg_payments?", "check this table"
Active incidentKeywords: alert, broken, stale, failing, incident, triage, wrong data"I have alerts firing", "data looks wrong", "something broke"
Coverage/monitoringKeywords: monitor, coverage, gaps, unmonitored, what should I watch"what should I monitor?", "where are my gaps?"
Agent instrumentationKeywords: instrument, set up tracing, set up Monte Carlo tracing, setting up an agent. Often mentions an AI framework (LangChain, LangGraph, OpenAI, Anthropic, CrewAI, Bedrock, SageMaker, Vertex AI)"instrument my agent", "set up MC tracing on my LangGraph agent", "setting up an agent"
General/exploratoryNo clear category, broad question"help me with data quality", "what can Monte Carlo do?"

Step 2: Gather scope (only if needed)

  • Specific asset known (from file context or user mention) → proceed to Step 3
  • Active incident, no scope → ask: "Want me to check recent alerts? Any specific time range or severity?"
  • Coverage/monitoring, no scope → ask: "Which warehouse should I look at, or should I check across all?"
  • General/exploratory → present the categories: "I can help with: (1) investigating active alerts or data issues, (2) analyzing monitoring coverage and creating monitors, or (3) checking the health of specific tables. What are you looking for?"

Step 3: Scoped API probe (when scope is available)

Only make API calls when you have enough context to scope them:

  • Specific asset → call get_alerts with the table's MCON or name filter, and get_monitors for that table
  • Active incident with scope → call get_alerts with the user's time range / severity filters
  • Coverage/monitoring → skip API probe, route directly to proactive monitoring workflow (it handles its own API calls)
  • If MCP tool calls fail → report the actual failure and missing evidence; use conversation intent only for routing, never infer healthy assets or configured authentication.

Always scope MCP calls tightly. Unscoped get_alerts, search, or get_monitors on large accounts can return hundreds of results, overflow the tool-result token limit, spill to disk, and force expensive chunk reads — burning user tokens and risking workflow failure. Minimum scoping:

  • get_alerts → time filter (created_after, default last 7 days) + at least one of warehouse, table_names, severity
  • search → needed to resolve a table name to its MCON (get_table requires MCON). ALWAYS pass limit (e.g. 5), the table name as query, and filter by warehouse_uuid or database/schema. warehouse_types alone ("snowflake") matches thousands of tables. Disambiguation rules when multiple matches return:
    1. If the user named a warehouse (e.g. "analytics-snowflake") → auto-pick the match whose warehouse_display_name matches and proceed. Do NOT stop to ask.
    2. If the user named a database/schema → auto-pick the match in that database/schema.
    3. If multiple matches remain, ask the user to choose the fully qualified asset. is_key_asset is importance metadata, not identity; warehouse or schema matches must also be unique.
  • get_monitors → always filter by mcons (table MCON) or warehouse_uuid

If you don't have enough scope, ask the user before calling.

Step 4: Route

Based on the combined signals from Steps 1-3:

Combined signalsConfidenceAction
Active alerts found + incident intentHighAuto-activate incident response workflow: read and follow ../incident-response/SKILL.md
Coverage intent + data project detectedHighAuto-activate proactive monitoring workflow: read and follow ../proactive-monitoring/SKILL.md
User asks to create a specific monitor (type + table known)HighAuto-activate monitoring-advisor: read and follow ../monitoring-advisor/SKILL.md
Table mentioned + "health" / "status" / "check" intentHighAuto-activate asset-health: read and follow ../asset-health/SKILL.md
Agent instrumentation intent (instrument / set up tracing / setting up an agent) + Python codebase contextHighAuto-activate instrument-agent: read and follow ../instrument-agent/SKILL.md
Ambiguous or conflicting signalsLowSuggest options and wait for user to choose

High confidence = auto-activate. Load the target skill's SKILL.md and begin executing it immediately. Do not ask for confirmation.

Low confidence = suggest. Present 2-3 options with brief descriptions and let the user choose. Example:

"Based on what you've described, I can:

  1. Investigate alerts — triage and fix active data issues (incident response workflow)
  2. Improve monitoring — find coverage gaps and create monitors (proactive monitoring workflow)

Which would be most helpful?"

Prevent guardrail

If the user is actively editing a dbt model file (making code changes, not just viewing or asking about it) and the prevent skill's hooks are active, do NOT route to any other skill. Instead respond:

"The prevent skill will automatically handle impact assessment for dbt model changes via its pre-edit hooks. No additional routing needed."

Examples

User: Apply this skill to my current task.
Assistant: Follow the workflow in this skill, cite limitations, and ask before risky steps.

Limitations

  • Imported upstream skill; verify credentials, permissions, and safety boundaries before execution.
  • Does not replace environment-specific validation, testing, or maintainer review.

Apache Spark Optimization

Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.

When to Use This Skill

  • Optimizing slow Spark jobs
  • Tuning memory and executor configuration
  • Implementing efficient partitioning strategies
  • Debugging Spark performance issues
  • Scaling Spark pipelines for large datasets
  • Reducing shuffle and data skew

Core Concepts

1. Spark Execution Model

Driver Program
    ↓
Job (triggered by action)
    ↓
Stages (separated by shuffles)
    ↓
Tasks (one per partition)

2. Key Performance Factors

FactorImpactSolution
ShuffleNetwork I/O, disk I/OMinimize wide transformations
Data SkewUneven task durationSalting, broadcast joins
SerializationCPU overheadUse Kryo, columnar formats
MemoryGC pressure, spillsTune executor memory
PartitionsParallelismRight-size partitions

Quick Start

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

# Create optimized Spark session
spark = (SparkSession.builder
    .appName("OptimizedJob")
    .config("spark.sql.adaptive.enabled", "true")
    .config("spark.sql.adaptive.coalescePartitions.enabled", "true")
    .config("spark.sql.adaptive.skewJoin.enabled", "true")
    .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
    .config("spark.sql.shuffle.partitions", "200")
    .getOrCreate())

# Read with optimized settings
df = (spark.read
    .format("parquet")
    .option("mergeSchema", "false")
    .load("s3://bucket/data/"))

# Efficient transformations
result = (df
    .filter(F.col("date") >= "2024-01-01")
    .select("id", "amount", "category")
    .groupBy("category")
    .agg(F.sum("amount").alias("total")))

result.write.mode("overwrite").parquet("s3://bucket/output/")

Detailed patterns and worked examples

Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.

Best Practices

Do's

  • Enable AQE - Adaptive query execution handles many issues
  • Use Parquet/Delta - Columnar formats with compression
  • Broadcast small tables - Avoid shuffle for small joins
  • Monitor Spark UI - Check for skew, spills, GC
  • Right-size partitions - 128MB - 256MB per partition

Don'ts

  • Don't collect large data - Keep data distributed
  • Don't use UDFs unnecessarily - Use built-in functions
  • Don't over-cache - Memory is limited
  • Don't ignore data skew - It dominates job time
  • Don't use .count() for existence - Use .take(1) or .isEmpty()

Sponsors

Details

Marketplace
marslanalvi/SkillsStack
Source
plugins/data/data-engineering-analytics
Category
Data
Keywords
data, data-engineering-analytics

More from marslanalvi/SkillsStack

All 45