
data-data-engineering-analytics
Data pipelines (Airflow, dbt, Spark), analytics, dashboards and metrics
- version
- 1.0.0
- GitHub stars
- 5
Install
/plugin install data-data-engineering-analytics@skills-stackRun it inside Claude Code after adding the marketplace with /plugin marketplace add marslanalvi/SkillsStack.
Contents
Discovered from sourceSkills 9
Apache Airflow DAG Patterns
Production-ready patterns for Apache Airflow including DAG design, operators, sensors, testing, and deployment strategies.
When to Use This Skill
- Creating data pipeline orchestration with Airflow
- Designing DAG structures and dependencies
- Implementing custom operators and sensors
- Testing Airflow DAGs locally
- Setting up Airflow in production
- Debugging failed DAG runs
Core Concepts
1. DAG Design Principles
| Principle | Description |
|---|---|
| Idempotent | Running twice produces same result |
| Atomic | Tasks succeed or fail completely |
| Incremental | Process only new/changed data |
| Observable | Logs, metrics, alerts at every step |
2. Task Dependencies
# Linear
task1 >> task2 >> task3
# Fan-out
task1 >> [task2, task3, task4]
# Fan-in
[task1, task2, task3] >> task4
# Complex
task1 >> task2 >> task4
task1 >> task3 >> task4
Quick Start
# dags/example_dag.py
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.operators.empty import EmptyOperator
default_args = {
'owner': 'data-team',
'depends_on_past': False,
'email_on_failure': True,
'email_on_retry': False,
'retries': 3,
'retry_delay': timedelta(minutes=5),
'retry_exponential_backoff': True,
'max_retry_delay': timedelta(hours=1),
}
with DAG(
dag_id='example_etl',
default_args=default_args,
description='Example ETL pipeline',
schedule='0 6 * * *', # Daily at 6 AM
start_date=datetime(2024, 1, 1),
catchup=False,
tags=['etl', 'example'],
max_active_runs=1,
) as dag:
start = EmptyOperator(task_id='start')
def extract_data(**context):
execution_date = context['ds']
# Extract logic here
return {'records': 1000}
extract = PythonOperator(
task_id='extract',
python_callable=extract_data,
)
end = EmptyOperator(task_id='end')
start >> extract >> end
Detailed patterns and worked examples
Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.
Best Practices
Do's
- Use TaskFlow API - Cleaner code, automatic XCom
- Set timeouts - Prevent zombie tasks
- Use
mode='reschedule'- For sensors, free up workers - Test DAGs - Unit tests and integration tests
- Idempotent tasks - Safe to retry
Don'ts
- Don't use
depends_on_past=True- Creates bottlenecks - Don't hardcode dates - Use
{{ ds }}macros - Don't use global state - Tasks should be stateless
- Don't skip catchup blindly - Understand implications
- Don't put heavy logic in DAG file - Import from modules
Calibrate First
Overview
A backtest result is only as good as the instrument that produced it. This skill
makes Claude measure the instrument before it measures the signal. It drives
assay-lite, a single-file,
offline Python tool. floor injects synthetic signals of known strength into the
user's own returns to find the smallest effect the data can detect. test charges
every run to an append-only registry before computing, and a result below the
floor is reported as UNDECIDABLE rather than "the signal doesn't work".
When to Use
- The user asks whether a trading signal, factor or ranking "works", is "significant", or has an "edge".
- The user wants to read an information coefficient (IC) or a cross-sectional backtest result, especially a null one.
- The user is about to try several variants of an idea and pick the best one.
Do not use it for single-asset timing signals, intraday data, live trading or position sizing. assay-lite does not cover them.
How It Works
Use only a previously reviewed local assay_lite.py at an explicit absolute path, verified against commit 78104a4356541affe2a469dfdf69cb1e418eb0c5, together with a fully locked and reviewed numpy/pandas/scipy environment. This skill does not download code or resolve dependencies. If absent, stop before execution and return the input schema and analysis plan. Do not substitute a same-named project file. A separate installation task must stage and inspect every file and dependency with installation hooks disabled, verify immutable bytes, then obtain approval before executing or writing the trial registry. uv caching is not isolation.
Four rules:
- Run
floorbefore interpreting any result. Until the floor is measured for these prices and this horizon, a non-significant result means nothing either way. - Never compare p with a conventional 0.05. The only bars are the ones the registry prints: 0.05 divided by every trial ever charged. "Bar for the last trial charged" and "the NEXT test will face" are different bars. Quote the one that matches the question.
- Trials are never refunded. Re-running a signal, tweaking a parameter or
trying another horizon is a new trial. Never suggest deleting registry lines or
using a fresh
--hometo reset the count. - Below the floor is UNDECIDABLE, not "no effect". Say "too small for this data to see", never "doesn't work".
Procedure:
- Check the inputs. Prices are
date,asset,close. The signal isdate,asset,signal, known at that date's close. - Run
breadth, and state the effective number of independent bets next to the asset count. - Run
floorfor the exact prices file and horizon you will test. - Write down in one sentence what you expect and why, then run
testonce. - Report the verdict line verbatim, with IC, p, the bar it faced, the dispersion ratio and the scaled floor.
Examples
python /absolute/reviewed/assay_lite.py demo # synthetic data; shows PASS, UNDECIDABLE, and a scaled floor
python /absolute/reviewed/assay_lite.py breadth --prices prices.csv
python /absolute/reviewed/assay_lite.py floor --prices prices.csv --horizon 1
python /absolute/reviewed/assay_lite.py test --prices prices.csv --signal signal.csv --name momentum_12_1 --horizon 1
python /absolute/reviewed/assay_lite.py registry
| verdict | say this | not this |
|---|---|---|
| PASS | "cleared the multiple-testing bar" | "proven", "profitable" |
| UNDECIDABLE | "too small for this data to see" | "doesn't work" |
| FAIL | "large but unreliable" | "zero" |
Limitations
- Cross-sectional and daily only, using Spearman IC. No timing signals and no portfolio simulation.
- No transaction costs. A PASS says nothing about profit after costs.
- Bonferroni is conservative when trials are correlated.
- The floor is calibrated with independent noise and then scaled up by each signal's daily-IC dispersion. That corrects for factor structure on average, not exactly.
- Nothing here is investment advice. A PASS is evidence of a ranking effect, not permission to trade.
Security & Safety Notes
- assay-lite makes no network calls and needs no API keys. It reads the CSVs the
user points it at and writes only to its own
.assay-lite/folder. - The registry is append-only on purpose. Don't edit or delete it on the user's behalf.
Data Quality Frameworks
Production patterns for implementing data quality with Great Expectations, dbt tests, and data contracts to ensure reliable data pipelines.
When to Use This Skill
- Implementing data quality checks in pipelines
- Setting up Great Expectations validation
- Building comprehensive dbt test suites
- Establishing data contracts between teams
- Monitoring data quality metrics
- Automating data validation in CI/CD
Core Concepts
1. Data Quality Dimensions
| Dimension | Description | Example Check |
|---|---|---|
| Completeness | No missing values | expect_column_values_to_not_be_null |
| Uniqueness | No duplicates | expect_column_values_to_be_unique |
| Validity | Values in expected range | expect_column_values_to_be_in_set |
| Accuracy | Data matches reality | Cross-reference validation |
| Consistency | No contradictions | expect_column_pair_values_A_to_be_greater_than_B |
| Timeliness | Data is recent | expect_column_max_to_be_between |
2. Testing Pyramid for Data
/\
/ \ Integration Tests (cross-table)
/────\
/ \ Unit Tests (single column)
/────────\
/ \ Schema Tests (structure)
/────────────\
Quick Start
Great Expectations Setup
# Install
pip install great_expectations
# Initialize project
great_expectations init
# Create datasource
great_expectations datasource new
# great_expectations/checkpoints/daily_validation.yml
import great_expectations as gx
# Create context
context = gx.get_context()
# Create expectation suite
suite = context.add_expectation_suite("orders_suite")
# Add expectations
suite.add_expectation(
gx.expectations.ExpectColumnValuesToNotBeNull(column="order_id")
)
suite.add_expectation(
gx.expectations.ExpectColumnValuesToBeUnique(column="order_id")
)
# Validate
results = context.run_checkpoint(checkpoint_name="daily_orders")
Detailed patterns and worked examples
Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.
Summary: {total_passed}/{total_tables} tables passed")
report.append("")
for table, result in results.items():
status = "✅" if result.passed else "❌"
report.append(f"### {status} {table}")
report.append(f"- Expectations: {result.total_expectations}")
report.append(f"- Failed: {result.failed_expectations}")
if not result.passed:
report.append("- Failed checks:")
for detail in result.details:
if not detail["success"]:
report.append(f" - {detail['expectation']}: {detail['observed_value']}")
report.append("")
return "\n".join(report)
Usage
context = gx.get_context() pipeline = DataQualityPipeline(context)
tables_to_validate = { "orders": "orders_suite", "customers": "customers_suite", "products": "products_suite", }
results = pipeline.run_all(tables_to_validate) report = pipeline.generate_report(results)
Fail pipeline if any table failed
if not all(r.passed for r in results.values()): print(report) raise ValueError("Data quality checks failed!")
## Best Practices
### Do's
- **Test early** - Validate source data before transformations
- **Test incrementally** - Add tests as you find issues
- **Document expectations** - Clear descriptions for each test
- **Alert on failures** - Integrate with monitoring
- **Version contracts** - Track schema changes
### Don'ts
- **Don't test everything** - Focus on critical columns
- **Don't ignore warnings** - They often precede failures
- **Don't skip freshness** - Stale data is bad data
- **Don't hardcode thresholds** - Use dynamic baselines
- **Don't test in isolation** - Test relationships too
Data Storytelling
Transform raw data into compelling narratives that drive decisions and inspire action.
When to Use This Skill
- Presenting analytics to executives
- Creating quarterly business reviews
- Building investor presentations
- Writing data-driven reports
- Communicating insights to non-technical audiences
- Making recommendations based on data
Core Concepts
1. Story Structure
Setup → Conflict → Resolution
Setup: Context and baseline
Conflict: The problem or opportunity
Resolution: Insights and recommendations
2. Narrative Arc
1. Hook: Grab attention with surprising insight
2. Context: Establish the baseline
3. Rising Action: Build through data points
4. Climax: The key insight
5. Resolution: Recommendations
6. Call to Action: Next steps
3. Three Pillars
| Pillar | Purpose | Components |
|---|---|---|
| Data | Evidence | Numbers, trends, comparisons |
| Narrative | Meaning | Context, causation, implications |
| Visuals | Clarity | Charts, diagrams, highlights |
Detailed patterns and worked examples
Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.
Best Practices
Do's
- Start with the "so what" - Lead with insight
- Use the rule of three - Three points, three comparisons
- Show, don't tell - Let data speak
- Make it personal - Connect to audience goals
- End with action - Clear next steps
Don'ts
- Don't data dump - Curate ruthlessly
- Don't bury the insight - Front-load key findings
- Don't use jargon - Match audience vocabulary
- Don't show methodology first - Context, then method
- Don't forget the narrative - Numbers need meaning
Data Throughput Accelerator
Use this skill when the bottleneck is moving, transforming, or saving lots of data. The goal is not just speed. The goal is faster correct data landing in the right place with proof.
First Distinction
Separate these before optimizing:
- source extraction speed;
- network transfer speed;
- warehouse/load speed;
- transform speed;
- serving-table freshness;
- live tail growth while the job runs.
A pipeline can be "fast" and still appear behind if new data arrives faster than the final catch-up window.
Fast Path Heuristics
- Move compute to where the data already is.
- Prefer warehouse-native scans, joins, and appends for large landed files.
- Use manifests or checkpoints so completed files/partitions are skipped.
- Use partitioning and clustering that match the read and append pattern.
- Batch small files, requests, and writes.
- Make writes idempotent through unique keys, manifests, or replaceable staging.
- Keep raw, derived, and serving tables separately accountable.
Workflow
- Read the current source, target, and manifest contracts.
- Measure backlog: external files, manifest rows, raw rows, derived rows, min/max timestamps, and unprocessed counts.
- Run a safe catch-up or sample benchmark.
- Compare variants: batch size, worker count, warehouse SQL, file grouping, staging shape, and manifest update method.
- Promote only the fastest path that keeps counts and timestamps coherent.
- Codify the path as a CLI, scheduled job, workflow, or runbook.
- Rerun final accounting after the codified path executes.
Accounting Output
Use a hard accounting block:
Data throughput result:
- Source files discovered: 294
- Files processed this run: 294
- Raw rows added: 9,683,598
- Derived rows added: 8,917,585
- Remaining tail: 24 files at readback time
- Runtime: 38.7s
- Correctness gate: manifest counts and table max timestamps match
Guardrails
- Do not delete raw data to make a metric look better.
- Do not skip failed files silently.
- Do not mix historical backfill status with live-tail freshness.
- Do not call a pipeline complete until the target tables and manifest agree.
- For finance, healthcare, regulated, or customer-impacting data, preserve replay evidence and approval gates.
dbt Transformation Patterns
Production-ready patterns for dbt (data build tool) including model organization, testing strategies, documentation, and incremental processing.
When to Use This Skill
- Building data transformation pipelines with dbt
- Organizing models into staging, intermediate, and marts layers
- Implementing data quality tests
- Creating incremental models for large datasets
- Documenting data models and lineage
- Setting up dbt project structure
Core Concepts
1. Model Layers (Medallion Architecture)
sources/ Raw data definitions
↓
staging/ 1:1 with source, light cleaning
↓
intermediate/ Business logic, joins, aggregations
↓
marts/ Final analytics tables
2. Naming Conventions
| Layer | Prefix | Example |
|---|---|---|
| Staging | stg_ | stg_stripe__payments |
| Intermediate | int_ | int_payments_pivoted |
| Marts | dim_, fct_ | dim_customers, fct_orders |
Quick Start
# dbt_project.yml
name: "analytics"
version: "1.0.0"
profile: "analytics"
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
vars:
start_date: "2020-01-01"
models:
analytics:
staging:
+materialized: view
+schema: staging
intermediate:
+materialized: ephemeral
marts:
+materialized: table
+schema: analytics
# Project structure
models/
├── staging/
│ ├── stripe/
│ │ ├── _stripe__sources.yml
│ │ ├── _stripe__models.yml
│ │ ├── stg_stripe__customers.sql
│ │ └── stg_stripe__payments.sql
│ └── shopify/
│ ├── _shopify__sources.yml
│ └── stg_shopify__orders.sql
├── intermediate/
│ └── finance/
│ └── int_payments_pivoted.sql
└── marts/
├── core/
│ ├── _core__models.yml
│ ├── dim_customers.sql
│ └── fct_orders.sql
└── finance/
└── fct_revenue.sql
Detailed patterns and worked examples
Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.
Best Practices
Do's
- Use staging layer - Clean data once, use everywhere
- Test aggressively - Not null, unique, relationships
- Document everything - Column descriptions, model descriptions
- Use incremental - For tables > 1M rows
- Version control - dbt project in Git
Don'ts
- Don't skip staging - Raw → mart is tech debt
- Don't hardcode dates - Use
{{ var('start_date') }} - Don't repeat logic - Extract to macros
- Don't test in prod - Use dev target
- Don't ignore freshness - Monitor source data
KPI Dashboard Design
Comprehensive patterns for designing effective Key Performance Indicator (KPI) dashboards that drive business decisions.
When to Use This Skill
- Designing executive dashboards
- Selecting meaningful KPIs
- Building real-time monitoring displays
- Creating department-specific metrics views
- Improving existing dashboard layouts
- Establishing metric governance
Core Concepts
1. KPI Framework
| Level | Focus | Update Frequency | Audience |
|---|---|---|---|
| Strategic | Long-term goals | Monthly/Quarterly | Executives |
| Tactical | Department goals | Weekly/Monthly | Managers |
| Operational | Day-to-day | Real-time/Daily | Teams |
2. SMART KPIs
Specific: Clear definition
Measurable: Quantifiable
Achievable: Realistic targets
Relevant: Aligned to goals
Time-bound: Defined period
3. Dashboard Hierarchy
├── Executive Summary (1 page)
│ ├── 4-6 headline KPIs
│ ├── Trend indicators
│ └── Key alerts
├── Department Views
│ ├── Sales Dashboard
│ ├── Marketing Dashboard
│ ├── Operations Dashboard
│ └── Finance Dashboard
└── Detailed Drilldowns
├── Individual metrics
└── Root cause analysis
Detailed worked examples and patterns
Detailed sections (starting with ## Common KPIs by Department) live in references/details.md. Read that file when the navigation summary above is insufficient.
Best Practices
Do's
- Limit to 5-7 KPIs - Focus on what matters
- Show context - Comparisons, trends, targets
- Use consistent colors - Red=bad, green=good
- Enable drilldown - From summary to detail
- Update appropriately - Match metric frequency
Don'ts
- Don't show vanity metrics - Focus on actionable data
- Don't overcrowd - White space aids comprehension
- Don't use 3D charts - They distort perception
- Don't hide methodology - Document calculations
- Don't ignore mobile - Ensure responsive design
Troubleshooting
MRR shown on dashboard contradicts finance's number
The most common cause is inconsistent treatment of annual plans. Finance may prorate to a daily rate while the dashboard normalizes to monthly. Align on a single formula and document it directly on the dashboard card:
-- Explicit formula shown in tooltip / data dictionary
-- Annual plans: divide total contract value by 12
-- Quarterly plans: divide by 3
-- Monthly plans: use as-is
CASE subscription_interval
WHEN 'monthly' THEN amount
WHEN 'quarterly' THEN amount / 3.0
WHEN 'yearly' THEN amount / 12.0
END AS normalized_mrr
Dashboard shows green but product team reports users complaining
The dashboard likely tracks system uptime (a lagging indicator) but not user-facing quality metrics. Add customer-perceived metrics alongside infrastructure metrics:
| Infrastructure (green) | User-perceived (add these) |
|---|---|
| API uptime 99.9% | P95 page load time |
| Error rate 0.1% | Task completion rate |
| Queue depth normal | Support ticket volume |
Retention cohort looks flat — no variation between cohorts
Check whether the cohort query is partitioning by signup month correctly. A common bug is using created_at::date instead of DATE_TRUNC('month', created_at), which groups by day and produces cohorts too small to show trends:
-- Wrong: too granular, cohorts are too small
DATE_TRUNC('day', created_at) AS cohort_date
-- Correct: monthly cohorts
DATE_TRUNC('month', created_at) AS cohort_month
Real-time dashboard hammers the database
A live dashboard refreshing every 10 seconds with complex cohort SQL will degrade production query performance. Separate OLAP workloads from OLTP by writing pre-aggregated metrics to a summary table via a scheduled job, and have the dashboard read from that:
# Scheduled every 5 minutes via cron/Celery
def refresh_mrr_summary():
conn.execute("""
INSERT INTO kpi_snapshot (metric, value, snapshot_at)
SELECT 'mrr', SUM(...), NOW()
FROM subscriptions WHERE status = 'active'
ON CONFLICT (metric) DO UPDATE SET value = EXCLUDED.value
""")
Alert thresholds fire constantly, team ignores them
Static thresholds set once and never reviewed cause alert fatigue. Use dynamic thresholds based on rolling averages so alerts fire only when the metric deviates significantly from its own baseline:
# Alert if current value is > 2 standard deviations from 30-day rolling mean
def is_anomalous(current: float, history: list[float]) -> bool:
mean = statistics.mean(history)
stdev = statistics.stdev(history)
return abs(current - mean) > 2 * stdev
Related Skills
data-storytelling- Turn dashboard findings into narratives that drive executive decisions
When to Use
- Use when this upstream workflow matches the user's stated goal.
- Use when the task requires the procedures documented in this skill.
Monte Carlo Context Detection
This skill determines which Monte Carlo skill or workflow best fits the user's current context. It activates reactively for ambiguous or multi-step data-related messages, gathers signals, and routes to the right skill or workflow.
Monte Carlo tool routing (required): Always call Monte Carlo MCP tools through this plugin's upstream bundled server when installed and verified for the intended account; its historical tool names are
mcp__plugin_mc-agent-toolkit_monte-carlo-mcp__<tool>(e.g.mcp__plugin_mc-agent-toolkit_monte-carlo-mcp__get_alerts). Bare tool names used in this skill (get_alerts,search,get_table, …) refer to that bundled server. If the session also has a separately-configuredmonte-carlo-mcpserver, do not route to it — it may point at a different endpoint or credentials.
This catalog tree does not bundle that server, routing CLAUDE.md, hooks, or companion skills. Verify availability and account identity; do not invent tools or silently use a different tenant. Routing does not authorize monitor creation, deployment, or other writes beyond the user's request.
Reference file for signal definitions: references/signal-definitions.md (relative to this file). Read it before routing.
When to activate this skill
This skill is activated by the CLAUDE.md routing table when:
- The user's message relates to data quality, alerts, incidents, coverage, or Monte Carlo — but doesn't clearly match a single skill in the routing table
- The user's intent is ambiguous or could span multiple skills
- The user asks a broad question like "help me with my data" or "what's going on?"
When NOT to activate this skill
- A skill or workflow is already active in the conversation — the active skill owns the conversation, do not intercept
- The user's message clearly matches a single skill in the CLAUDE.md routing table — route directly, no need for context detection
- The user is editing a dbt model — defer to the
preventskill only when its hooks are verified active - The user's message is not data-related at all
Workflow: Reactive Routing
This skill is purely reactive — it activates for ambiguous or multi-step data-related messages and routes them.
Follow these steps in order.
Step 0: Fast-path clear intent (stop early if matched)
Before doing anything else, check whether the user's message unambiguously matches a single existing skill. If so, skip the rest of this workflow and immediately load that skill — do NOT read references/signal-definitions.md, do NOT make API probes.
| Clear user intent | Skill to load immediately |
|---|---|
| "Check health of [named table]" / "status of [named table]" | ../asset-health/SKILL.md |
| "Create a [monitor type] on [named table]" | ../monitoring-advisor/SKILL.md |
| "Investigate alert on [named table]" / "why is [named table] stale/broken?" | ../incident-response/SKILL.md |
| "What should I monitor?" / "where are my coverage gaps?" | ../proactive-monitoring/SKILL.md |
| "Instrument my agent" / "set up Monte Carlo tracing on [named framework] agent" / "setting up an agent" | ../instrument-agent/SKILL.md |
Context-detection is for ambiguous requests only. If the request is clear, routing through this skill wastes turns and tokens.
If no clear match, proceed to Step 1.
Step 1: Categorize intent
Read references/signal-definitions.md for the full signal catalog. Determine which category the user's message falls into:
| Category | Signals | Example messages |
|---|---|---|
| Specific asset | User mentions a table name, or has a .sql model file open in their IDE | "what's wrong with stg_payments?", "check this table" |
| Active incident | Keywords: alert, broken, stale, failing, incident, triage, wrong data | "I have alerts firing", "data looks wrong", "something broke" |
| Coverage/monitoring | Keywords: monitor, coverage, gaps, unmonitored, what should I watch | "what should I monitor?", "where are my gaps?" |
| Agent instrumentation | Keywords: instrument, set up tracing, set up Monte Carlo tracing, setting up an agent. Often mentions an AI framework (LangChain, LangGraph, OpenAI, Anthropic, CrewAI, Bedrock, SageMaker, Vertex AI) | "instrument my agent", "set up MC tracing on my LangGraph agent", "setting up an agent" |
| General/exploratory | No clear category, broad question | "help me with data quality", "what can Monte Carlo do?" |
Step 2: Gather scope (only if needed)
- Specific asset known (from file context or user mention) → proceed to Step 3
- Active incident, no scope → ask: "Want me to check recent alerts? Any specific time range or severity?"
- Coverage/monitoring, no scope → ask: "Which warehouse should I look at, or should I check across all?"
- General/exploratory → present the categories: "I can help with: (1) investigating active alerts or data issues, (2) analyzing monitoring coverage and creating monitors, or (3) checking the health of specific tables. What are you looking for?"
Step 3: Scoped API probe (when scope is available)
Only make API calls when you have enough context to scope them:
- Specific asset → call
get_alertswith the table's MCON or name filter, andget_monitorsfor that table - Active incident with scope → call
get_alertswith the user's time range / severity filters - Coverage/monitoring → skip API probe, route directly to proactive monitoring workflow (it handles its own API calls)
- If MCP tool calls fail → report the actual failure and missing evidence; use conversation intent only for routing, never infer healthy assets or configured authentication.
Always scope MCP calls tightly. Unscoped get_alerts, search, or get_monitors on large accounts can return hundreds of results, overflow the tool-result token limit, spill to disk, and force expensive chunk reads — burning user tokens and risking workflow failure. Minimum scoping:
get_alerts→ time filter (created_after, default last 7 days) + at least one ofwarehouse,table_names,severitysearch→ needed to resolve a table name to its MCON (get_tablerequires MCON). ALWAYS passlimit(e.g. 5), the table name asquery, and filter bywarehouse_uuidordatabase/schema.warehouse_typesalone ("snowflake") matches thousands of tables. Disambiguation rules when multiple matches return:- If the user named a warehouse (e.g. "analytics-snowflake") → auto-pick the match whose
warehouse_display_namematches and proceed. Do NOT stop to ask. - If the user named a database/schema → auto-pick the match in that database/schema.
- If multiple matches remain, ask the user to choose the fully qualified asset.
is_key_assetis importance metadata, not identity; warehouse or schema matches must also be unique.
- If the user named a warehouse (e.g. "analytics-snowflake") → auto-pick the match whose
get_monitors→ always filter bymcons(table MCON) orwarehouse_uuid
If you don't have enough scope, ask the user before calling.
Step 4: Route
Based on the combined signals from Steps 1-3:
| Combined signals | Confidence | Action |
|---|---|---|
| Active alerts found + incident intent | High | Auto-activate incident response workflow: read and follow ../incident-response/SKILL.md |
| Coverage intent + data project detected | High | Auto-activate proactive monitoring workflow: read and follow ../proactive-monitoring/SKILL.md |
| User asks to create a specific monitor (type + table known) | High | Auto-activate monitoring-advisor: read and follow ../monitoring-advisor/SKILL.md |
| Table mentioned + "health" / "status" / "check" intent | High | Auto-activate asset-health: read and follow ../asset-health/SKILL.md |
| Agent instrumentation intent (instrument / set up tracing / setting up an agent) + Python codebase context | High | Auto-activate instrument-agent: read and follow ../instrument-agent/SKILL.md |
| Ambiguous or conflicting signals | Low | Suggest options and wait for user to choose |
High confidence = auto-activate. Load the target skill's SKILL.md and begin executing it immediately. Do not ask for confirmation.
Low confidence = suggest. Present 2-3 options with brief descriptions and let the user choose. Example:
"Based on what you've described, I can:
- Investigate alerts — triage and fix active data issues (incident response workflow)
- Improve monitoring — find coverage gaps and create monitors (proactive monitoring workflow)
Which would be most helpful?"
Prevent guardrail
If the user is actively editing a dbt model file (making code changes, not just viewing or asking about it) and the prevent skill's hooks are active, do NOT route to any other skill. Instead respond:
"The prevent skill will automatically handle impact assessment for dbt model changes via its pre-edit hooks. No additional routing needed."
Examples
User: Apply this skill to my current task.
Assistant: Follow the workflow in this skill, cite limitations, and ask before risky steps.
Limitations
- Imported upstream skill; verify credentials, permissions, and safety boundaries before execution.
- Does not replace environment-specific validation, testing, or maintainer review.
Apache Spark Optimization
Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.
When to Use This Skill
- Optimizing slow Spark jobs
- Tuning memory and executor configuration
- Implementing efficient partitioning strategies
- Debugging Spark performance issues
- Scaling Spark pipelines for large datasets
- Reducing shuffle and data skew
Core Concepts
1. Spark Execution Model
Driver Program
↓
Job (triggered by action)
↓
Stages (separated by shuffles)
↓
Tasks (one per partition)
2. Key Performance Factors
| Factor | Impact | Solution |
|---|---|---|
| Shuffle | Network I/O, disk I/O | Minimize wide transformations |
| Data Skew | Uneven task duration | Salting, broadcast joins |
| Serialization | CPU overhead | Use Kryo, columnar formats |
| Memory | GC pressure, spills | Tune executor memory |
| Partitions | Parallelism | Right-size partitions |
Quick Start
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
# Create optimized Spark session
spark = (SparkSession.builder
.appName("OptimizedJob")
.config("spark.sql.adaptive.enabled", "true")
.config("spark.sql.adaptive.coalescePartitions.enabled", "true")
.config("spark.sql.adaptive.skewJoin.enabled", "true")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config("spark.sql.shuffle.partitions", "200")
.getOrCreate())
# Read with optimized settings
df = (spark.read
.format("parquet")
.option("mergeSchema", "false")
.load("s3://bucket/data/"))
# Efficient transformations
result = (df
.filter(F.col("date") >= "2024-01-01")
.select("id", "amount", "category")
.groupBy("category")
.agg(F.sum("amount").alias("total")))
result.write.mode("overwrite").parquet("s3://bucket/output/")
Detailed patterns and worked examples
Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.
Best Practices
Do's
- Enable AQE - Adaptive query execution handles many issues
- Use Parquet/Delta - Columnar formats with compression
- Broadcast small tables - Avoid shuffle for small joins
- Monitor Spark UI - Check for skew, spills, GC
- Right-size partitions - 128MB - 256MB per partition
Don'ts
- Don't collect large data - Keep data distributed
- Don't use UDFs unnecessarily - Use built-in functions
- Don't over-cache - Memory is limited
- Don't ignore data skew - It dominates job time
- Don't use
.count()for existence - Use.take(1)or.isEmpty()
Sponsors
Details
- Marketplace
- marslanalvi/SkillsStack
- Source
- plugins/data/data-engineering-analytics
- Category
- Data
- Keywords
- data, data-engineering-analytics
More from marslanalvi/SkillsStack
All 45- data-researchv1.0.0
- design-accessibilityv1.0.0
- design-media-creativev1.0.0
- design-ui-uxv1.0.0
- devops-ci-cdv1.0.0
- devops-cloudv1.0.0
- devops-containers-kubernetesv1.0.0
- devops-infrastructure-networkingv1.0.0
- devops-observability-srev1.0.0
- engineering-architecture-apiv1.0.0
- engineering-code-review-qualityv1.0.0
- engineering-databasesv1.0.0
- engineering-debuggingv1.0.0
- engineering-dev-workflowv1.0.0
- engineering-frontendv1.0.0
- engineering-game-developmentv1.0.0
- engineering-git-githubv1.0.0
- engineering-javascript-typescriptv1.0.0
- engineering-jvmv1.0.0
- engineering-mobilev1.0.0
- engineering-other-languagesv1.0.0
- engineering-pythonv1.0.0
- engineering-systems-languagesv1.0.0
- engineering-testingv1.0.0