
engineering-python
Python patterns, packaging, typing, async, testing and frameworks (Django, FastAPI)
- version
- 1.0.0
- GitHub stars
- 5
Install
/plugin install engineering-python@skills-stackRun it inside Claude Code after adding the marketplace with /plugin marketplace add marslanalvi/SkillsStack.
Contents
Discovered from sourceSkills 29
Async Python Patterns
Comprehensive guidance for implementing asynchronous Python applications using asyncio, concurrent programming patterns, and async/await for building high-performance, non-blocking systems.
When to Use This Skill
- Building async web APIs (FastAPI, aiohttp, Sanic)
- Implementing concurrent I/O operations (database, file, network)
- Creating web scrapers with concurrent requests
- Developing real-time applications (WebSocket servers, chat systems)
- Processing multiple independent tasks simultaneously
- Building microservices with async communication
- Optimizing I/O-bound workloads
- Implementing async background tasks and queues
Sync vs Async Decision Guide
Before adopting async, consider whether it's the right choice for your use case.
| Use Case | Recommended Approach |
|---|---|
| Many concurrent network/DB calls | asyncio |
| CPU-bound computation | multiprocessing or thread pool |
| Mixed I/O + CPU | Offload CPU work with asyncio.to_thread() |
| Simple scripts, few connections | Sync (simpler, easier to debug) |
| Web APIs with high concurrency | Async frameworks (FastAPI, aiohttp) |
Key Rule: Stay fully sync or fully async within a call path. Mixing creates hidden blocking and complexity.
Core Concepts
1. Event Loop
The event loop is the heart of asyncio, managing and scheduling asynchronous tasks.
Key characteristics:
- Single-threaded cooperative multitasking
- Schedules coroutines for execution
- Handles I/O operations without blocking
- Manages callbacks and futures
2. Coroutines
Functions defined with async def that can be paused and resumed.
Syntax:
async def my_coroutine():
result = await some_async_operation()
return result
3. Tasks
Scheduled coroutines that run concurrently on the event loop.
4. Futures
Low-level objects representing eventual results of async operations.
5. Async Context Managers
Resources that support async with for proper cleanup.
6. Async Iterators
Objects that support async for for iterating over async data sources.
Quick Start
import asyncio
async def main():
print("Hello")
await asyncio.sleep(1)
print("World")
# Python 3.7+
asyncio.run(main())
Fundamental Patterns
Pattern 1: Basic Async/Await
import asyncio
async def fetch_data(url: str) -> dict:
"""Fetch data from URL asynchronously."""
await asyncio.sleep(1) # Simulate I/O
return {"url": url, "data": "result"}
async def main():
result = await fetch_data("https://api.example.com")
print(result)
asyncio.run(main())
Pattern 2: Concurrent Execution with gather()
import asyncio
from typing import List
async def fetch_user(user_id: int) -> dict:
"""Fetch user data."""
await asyncio.sleep(0.5)
return {"id": user_id, "name": f"User {user_id}"}
async def fetch_all_users(user_ids: List[int]) -> List[dict]:
"""Fetch multiple users concurrently."""
tasks = [fetch_user(uid) for uid in user_ids]
results = await asyncio.gather(*tasks)
return results
async def main():
user_ids = [1, 2, 3, 4, 5]
users = await fetch_all_users(user_ids)
print(f"Fetched {len(users)} users")
asyncio.run(main())
Pattern 3: Task Creation and Management
import asyncio
async def background_task(name: str, delay: int):
"""Long-running background task."""
print(f"{name} started")
await asyncio.sleep(delay)
print(f"{name} completed")
return f"Result from {name}"
async def main():
# Create tasks
task1 = asyncio.create_task(background_task("Task 1", 2))
task2 = asyncio.create_task(background_task("Task 2", 1))
# Do other work
print("Main: doing other work")
await asyncio.sleep(0.5)
# Wait for tasks
result1 = await task1
result2 = await task2
print(f"Results: {result1}, {result2}")
asyncio.run(main())
Pattern 4: Error Handling in Async Code
import asyncio
from typing import List, Optional
async def risky_operation(item_id: int) -> dict:
"""Operation that might fail."""
await asyncio.sleep(0.1)
if item_id % 3 == 0:
raise ValueError(f"Item {item_id} failed")
return {"id": item_id, "status": "success"}
async def safe_operation(item_id: int) -> Optional[dict]:
"""Wrapper with error handling."""
try:
return await risky_operation(item_id)
except ValueError as e:
print(f"Error: {e}")
return None
async def process_items(item_ids: List[int]):
"""Process multiple items with error handling."""
tasks = [safe_operation(iid) for iid in item_ids]
results = await asyncio.gather(*tasks, return_exceptions=True)
# Filter out failures
successful = [r for r in results if r is not None and not isinstance(r, Exception)]
failed = [r for r in results if r is None or isinstance(r, Exception)]
print(f"Success: {len(successful)}, Failed: {len(failed)}")
return successful
asyncio.run(process_items([1, 2, 3, 4, 5, 6]))
Pattern 5: Timeout Handling
import asyncio
async def slow_operation(delay: int) -> str:
"""Operation that takes time."""
await asyncio.sleep(delay)
return f"Completed after {delay}s"
async def with_timeout():
"""Execute operation with timeout."""
try:
result = await asyncio.wait_for(slow_operation(5), timeout=2.0)
print(result)
except asyncio.TimeoutError:
print("Operation timed out")
asyncio.run(with_timeout())
Detailed worked examples and patterns
Detailed sections (starting with ## Advanced Patterns) live in references/details.md. Read that file when the navigation summary above is insufficient.
Common Pitfalls
1. Forgetting await
# Wrong - returns coroutine object, doesn't execute
result = async_function()
# Correct
result = await async_function()
2. Blocking the Event Loop
# Wrong - blocks event loop
import time
async def bad():
time.sleep(1) # Blocks!
# Correct
async def good():
await asyncio.sleep(1) # Non-blocking
3. Not Handling Cancellation
async def cancelable_task():
"""Task that handles cancellation."""
try:
while True:
await asyncio.sleep(1)
print("Working...")
except asyncio.CancelledError:
print("Task cancelled, cleaning up...")
# Perform cleanup
raise # Re-raise to propagate cancellation
4. Mixing Sync and Async Code
# Wrong - can't call async from sync directly
def sync_function():
result = await async_function() # SyntaxError!
# Correct
def sync_function():
result = asyncio.run(async_function())
Testing Async Code
import asyncio
import pytest
# Using pytest-asyncio
@pytest.mark.asyncio
async def test_async_function():
"""Test async function."""
result = await fetch_data("https://api.example.com")
assert result is not None
@pytest.mark.asyncio
async def test_with_timeout():
"""Test with timeout."""
with pytest.raises(asyncio.TimeoutError):
await asyncio.wait_for(slow_operation(5), timeout=1.0)
DBOS Python Best Practices
Guide for building reliable, fault-tolerant Python applications with DBOS durable workflows.
When to Use
Reference these guidelines when:
- Adding DBOS to existing Python code
- Creating workflows and steps
- Using queues for concurrency control
- Implementing workflow communication (events, messages, streams)
- Configuring and launching DBOS applications
- Using DBOSClient from external applications
- Testing DBOS applications
Runtime boundaries
Examples are version-sensitive fragments with omitted app setup/imports/handlers; verify installed DBOS APIs. Durable workflow identity does not make arbitrary external side effects exactly-once: steps may repeat after a crash before checkpointing. Use provider idempotency keys plus reconciliation for payments/messages. Authorize database targets and writes, protect serialized inputs/results, and never deserialize untrusted pickle data. API routes need authentication, tenant ownership, webhook signature/replay validation and bounded waits. Destructive test reset must use a dedicated disposable system database; a variable name alone is not target proof.
Rule Categories by Priority
| Priority | Category | Impact | Prefix |
|---|---|---|---|
| 1 | Lifecycle | CRITICAL | lifecycle- |
| 2 | Workflow | CRITICAL | workflow- |
| 3 | Step | HIGH | step- |
| 4 | Queue | HIGH | queue- |
| 5 | Communication | MEDIUM | comm- |
| 6 | Pattern | MEDIUM | pattern- |
| 7 | Testing | LOW-MEDIUM | test- |
| 8 | Client | MEDIUM | client- |
| 9 | Advanced | LOW | advanced- |
Critical Rules
DBOS Configuration and Launch
A DBOS application MUST configure and launch DBOS inside its main function:
import os
from dbos import DBOS, DBOSConfig
@DBOS.workflow()
def my_workflow():
pass
if __name__ == "__main__":
config: DBOSConfig = {
"name": "my-app",
"system_database_url": os.environ.get("DBOS_SYSTEM_DATABASE_URL"),
}
DBOS(config=config)
DBOS.launch()
Workflow and Step Structure
Workflows are comprised of steps. Any function performing complex operations or accessing external services must be a step:
@DBOS.step()
def call_external_api():
return requests.get("https://api.example.com").json()
@DBOS.workflow()
def my_workflow():
result = call_external_api()
return result
Key Constraints
- Do NOT call
DBOS.start_workfloworDBOS.recvfrom a step - Do NOT use threads to start workflows - use
DBOS.start_workflowor queues - Workflows MUST be deterministic - non-deterministic operations go in steps
- Do NOT create/update global variables from workflows or steps
How to Use
Read individual rule files for detailed explanations and examples:
references/lifecycle-config.md
references/workflow-determinism.md
references/queue-concurrency.md
References
Limitations
- Use this skill only when the task clearly matches the scope described above.
- Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
- Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.
Django + Celery Async Task Patterns
Production-grade patterns for background task processing in Django using Celery with Redis or RabbitMQ.
When to Activate
- Adding background jobs or async processing to a Django app
- Implementing periodic/scheduled tasks
- Offloading slow operations (email, PDF generation, API calls) from request cycle
- Setting up Celery Beat for cron-like scheduling
- Debugging task failures, retries, or queue backlogs
- Writing tests for Celery tasks
Project Setup
Installation
pip install 'celery[redis]' django-celery-results django-celery-beat
celery.py — App Entrypoint
# config/celery.py
import os
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings.development')
app = Celery('myproject')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks() # Discovers tasks.py in each INSTALLED_APP
@app.task(bind=True, ignore_result=True)
def debug_task(self):
print(f'Request: {self.request!r}')
# config/__init__.py
from .celery import app as celery_app
__all__ = ('celery_app',)
Django Settings
# config/settings/base.py
# Broker (Redis recommended for production)
CELERY_BROKER_URL = env('CELERY_BROKER_URL', default='redis://localhost:6379/0')
CELERY_RESULT_BACKEND = env('CELERY_RESULT_BACKEND', default='django-db')
# Serialization
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
# Task behavior
CELERY_TASK_TRACK_STARTED = True
CELERY_TASK_TIME_LIMIT = 30 * 60 # Hard limit: 30 min
CELERY_TASK_SOFT_TIME_LIMIT = 25 * 60 # Soft limit: sends SoftTimeLimitExceeded
CELERY_WORKER_PREFETCH_MULTIPLIER = 1 # Prevent worker hoarding long tasks
CELERY_TASK_ACKS_LATE = True # Re-queue on worker crash
# Result persistence
CELERY_RESULT_EXPIRES = 60 * 60 * 24 # Keep results 24 hours
# Beat scheduler (for periodic tasks)
CELERY_BEAT_SCHEDULER = 'django_celery_beat.schedulers:DatabaseScheduler'
# Installed apps
INSTALLED_APPS += [
'django_celery_results',
'django_celery_beat',
]
Running Workers
# Start worker (development)
celery -A config worker --loglevel=info
# Start beat scheduler (periodic tasks)
celery -A config beat --loglevel=info --scheduler django_celery_beat.schedulers:DatabaseScheduler
# Combined worker + beat (dev only, never production)
celery -A config worker --beat --loglevel=info
# Production: multiple workers with concurrency
celery -A config worker --loglevel=warning --concurrency=4 -Q default,high_priority
Task Design Patterns
Basic Task
# apps/notifications/tasks.py
from celery import shared_task
import logging
logger = logging.getLogger(__name__)
@shared_task(name='notifications.send_welcome_email')
def send_welcome_email(user_id: int) -> None:
"""Send welcome email to newly registered user."""
from apps.users.models import User
from apps.notifications.services import EmailService
try:
user = User.objects.get(pk=user_id)
except User.DoesNotExist:
logger.warning('send_welcome_email: user %s not found', user_id)
return # Idempotent — do not raise, task already impossible to complete
EmailService.send_welcome(user)
logger.info('Welcome email sent to user %s', user_id)
Retryable Task
@shared_task(
bind=True,
name='integrations.sync_to_crm',
max_retries=5,
default_retry_delay=60, # seconds before first retry
autoretry_for=(ConnectionError, TimeoutError),
retry_backoff=True, # exponential backoff
retry_backoff_max=600, # cap at 10 minutes
retry_jitter=True, # randomise to avoid thundering herd
)
def sync_contact_to_crm(self, contact_id: int) -> dict:
"""Sync contact to external CRM with retry on transient failures."""
from apps.crm.services import CRMClient
try:
result = CRMClient().sync(contact_id)
return result
except CRMClient.RateLimitError as exc:
# Specific retry delay from response header
raise self.retry(exc=exc, countdown=int(exc.retry_after))
Idempotent Task Pattern
Design tasks so they can safely run multiple times with the same inputs:
@shared_task(name='orders.mark_shipped')
def mark_order_shipped(order_id: int, tracking_number: str) -> None:
"""Mark order as shipped — safe to run multiple times."""
from apps.orders.models import Order
updated = Order.objects.filter(
pk=order_id,
status=Order.Status.PROCESSING, # Guard: only update if not already shipped
).update(
status=Order.Status.SHIPPED,
tracking_number=tracking_number,
)
if not updated:
logger.info('mark_order_shipped: order %s already shipped or not found', order_id)
Task with Soft Time Limit
from celery.exceptions import SoftTimeLimitExceeded
@shared_task(
bind=True,
name='reports.generate_pdf',
soft_time_limit=120,
time_limit=150,
)
def generate_pdf_report(self, report_id: int) -> str:
"""Generate PDF report with graceful timeout handling."""
from apps.reports.services import PDFGenerator
try:
path = PDFGenerator.build(report_id)
return path
except SoftTimeLimitExceeded:
# Clean up partial files before hard kill
PDFGenerator.cleanup(report_id)
raise
Calling Tasks
from datetime import timedelta
from django.utils import timezone
# Fire and forget (async)
send_welcome_email.delay(user.pk)
# Schedule in the future
send_reminder.apply_async(args=[user.pk], countdown=3600) # 1 hour from now
send_reminder.apply_async(args=[user.pk], eta=timezone.now() + timedelta(days=1))
# Apply with queue routing
sync_contact_to_crm.apply_async(args=[contact.pk], queue='high_priority')
# Run synchronously (tests / debugging only)
result = generate_pdf_report.apply(args=[report.pk])
Beat Scheduling (Periodic Tasks)
Code-Defined Schedule
# config/settings/base.py
from celery.schedules import crontab
CELERY_BEAT_SCHEDULE = {
'cleanup-expired-sessions': {
'task': 'users.cleanup_expired_sessions',
'schedule': crontab(hour=2, minute=0), # 2am daily
},
'sync-inventory': {
'task': 'products.sync_inventory',
'schedule': 60.0, # every 60 seconds
},
'weekly-digest': {
'task': 'notifications.send_weekly_digest',
'schedule': crontab(day_of_week='monday', hour=8, minute=0),
},
}
Database-Defined Schedule (via django-celery-beat)
# Manage periodic tasks from Django admin or code
from django_celery_beat.models import PeriodicTask, CrontabSchedule
import json
schedule, _ = CrontabSchedule.objects.get_or_create(
hour='*/6', minute='0',
timezone='UTC',
)
PeriodicTask.objects.update_or_create(
name='Sync inventory every 6 hours',
defaults={
'crontab': schedule,
'task': 'products.sync_inventory',
'args': json.dumps([]),
'enabled': True,
}
)
Canvas: Chaining and Grouping Tasks
from celery import chain, group, chord
# Chain: run tasks sequentially, passing results
pipeline = chain(
fetch_data.s(source_id),
transform_data.s(), # receives fetch_data result as first arg
load_to_warehouse.s(),
)
pipeline.delay()
# Group: run tasks in parallel
parallel = group(
send_welcome_email.s(user_id)
for user_id in new_user_ids
)
parallel.delay()
# Chord: parallel tasks + callback when all complete
result = chord(
group(process_chunk.s(chunk) for chunk in data_chunks),
aggregate_results.s(), # called with list of chunk results
)
result.delay()
Error Handling and Dead Letter Queue
# apps/core/tasks.py
from celery.signals import task_failure
@task_failure.connect
def on_task_failure(sender, task_id, exception, args, kwargs, traceback, einfo, **kw):
"""Log all task failures to Sentry / alerting."""
import sentry_sdk
with sentry_sdk.new_scope() as scope:
scope.set_context('celery', {
'task': sender.name,
'task_id': task_id,
'args': args,
'kwargs': kwargs,
})
sentry_sdk.capture_exception(exception)
# Route failed tasks to dead-letter queue after max retries
@shared_task(
bind=True,
max_retries=3,
name='payments.charge_card',
)
def charge_card(self, order_id: int) -> None:
from apps.payments.models import Order, FailedCharge
try:
_do_charge(order_id)
except Exception as exc:
if self.request.retries >= self.max_retries:
# Persist to dead-letter table for manual review
FailedCharge.objects.create(
order_id=order_id,
error=str(exc),
task_id=self.request.id,
)
return # Don't raise — task is permanently failed
raise self.retry(exc=exc)
Testing Celery Tasks
Unit Testing (No Broker)
# tests/test_tasks.py
import pytest
from unittest.mock import patch, MagicMock
from apps.notifications.tasks import send_welcome_email
class TestSendWelcomeEmail:
@pytest.mark.django_db
def test_sends_email_to_existing_user(self, user):
with patch('apps.notifications.services.EmailService') as mock_email:
send_welcome_email(user.pk)
mock_email.send_welcome.assert_called_once_with(user)
@pytest.mark.django_db
def test_skips_missing_user_gracefully(self):
"""Should not raise when user is deleted between enqueue and execute."""
send_welcome_email(99999) # Non-existent user — must not raise
Integration Testing with CELERY_TASK_ALWAYS_EAGER
# config/settings/test.py
CELERY_TASK_ALWAYS_EAGER = True # Run tasks synchronously in tests
CELERY_TASK_EAGER_PROPAGATES = True # Re-raise exceptions from tasks
# tests/test_integration.py
@pytest.mark.django_db
def test_registration_triggers_welcome_email(client):
with patch('apps.notifications.services.EmailService') as mock_email:
response = client.post('/api/users/', {
'email': 'new@example.com',
'password': 'strongpass123',
})
assert response.status_code == 201
mock_email.send_welcome.assert_called_once()
Testing Retries
@pytest.mark.django_db
def test_task_retries_on_connection_error():
with patch('apps.crm.services.CRMClient.sync') as mock_sync:
mock_sync.side_effect = ConnectionError('timeout')
with pytest.raises(ConnectionError):
sync_contact_to_crm.apply(args=[1], throw=True)
assert mock_sync.call_count == 1 # First attempt only when eager
Monitoring
# Inspect active workers and queues
celery -A config inspect active
celery -A config inspect stats
celery -A config inspect reserved
# Check queue lengths (Redis)
redis-cli llen celery
# Flower: web-based real-time monitor
pip install flower
celery -A config flower --port=5555
Anti-Patterns
# BAD: Passing model instances — they may be stale by execution time
send_welcome_email.delay(user) # Never pass ORM objects
send_welcome_email.delay(user.pk) # Always pass PKs
# BAD: Calling tasks synchronously in production views
result = generate_report.apply() # Blocks the request thread
# BAD: Non-idempotent task without guards
@shared_task
def charge_and_fulfill(order_id):
order.charge() # May charge twice if task retries!
order.fulfill()
# GOOD: Idempotent with status guard
@shared_task
def charge_and_fulfill(order_id):
order = Order.objects.select_for_update().get(pk=order_id)
if order.status != Order.Status.PENDING:
return # Already processed
order.charge()
order.fulfill()
Production Checklist
| Check | Setting |
|---|---|
| Worker restarts on crash | supervisord or systemd unit |
CELERY_TASK_ACKS_LATE = True | Re-queue tasks on worker crash |
CELERY_WORKER_PREFETCH_MULTIPLIER = 1 | Fair distribution of long tasks |
| Separate queues per priority | -Q default,high_priority,low_priority |
CELERY_TASK_SOFT_TIME_LIMIT set | Graceful timeout before hard kill |
| Sentry integration | Capture all task_failure signals |
| Flower or other monitor | Visibility into queue depths |
| Beat runs on single node only | Prevents duplicate scheduled task execution |
Related Skills
django-patterns— ORM, service layer, and project structuredjango-tdd— Testing Django models, views, and servicespython-testing— pytest configuration and fixtures
Django Development Patterns
Production-grade Django architecture patterns for scalable, maintainable applications.
When to Activate
- Building Django web applications
- Designing Django REST Framework APIs
- Working with Django ORM and models
- Setting up Django project structure
- Implementing caching, signals, middleware
Project Structure
Recommended Layout
myproject/
├── config/
│ ├── __init__.py
│ ├── settings/
│ │ ├── __init__.py
│ │ ├── base.py # Base settings
│ │ ├── development.py # Dev settings
│ │ ├── production.py # Production settings
│ │ └── test.py # Test settings
│ ├── urls.py
│ ├── wsgi.py
│ └── asgi.py
├── manage.py
└── apps/
├── __init__.py
├── users/
│ ├── __init__.py
│ ├── models.py
│ ├── views.py
│ ├── serializers.py
│ ├── urls.py
│ ├── permissions.py
│ ├── filters.py
│ ├── services.py
│ └── tests/
└── products/
└── ...
Split Settings Pattern
# config/settings/base.py
from pathlib import Path
BASE_DIR = Path(__file__).resolve().parent.parent.parent
SECRET_KEY = env('DJANGO_SECRET_KEY')
DEBUG = False
ALLOWED_HOSTS = []
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'rest_framework',
'rest_framework.authtoken',
'corsheaders',
# Local apps
'apps.users',
'apps.products',
]
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'whitenoise.middleware.WhiteNoiseMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'corsheaders.middleware.CorsMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
]
ROOT_URLCONF = 'config.urls'
WSGI_APPLICATION = 'config.wsgi.application'
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': env('DB_NAME'),
'USER': env('DB_USER'),
'PASSWORD': env('DB_PASSWORD'),
'HOST': env('DB_HOST'),
'PORT': env('DB_PORT', default='5432'),
}
}
# config/settings/development.py
from .base import *
DEBUG = True
ALLOWED_HOSTS = ['localhost', '127.0.0.1']
DATABASES['default']['NAME'] = 'myproject_dev'
INSTALLED_APPS += ['debug_toolbar']
MIDDLEWARE += ['debug_toolbar.middleware.DebugToolbarMiddleware']
EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'
# config/settings/production.py
from .base import *
DEBUG = False
ALLOWED_HOSTS = env.list('ALLOWED_HOSTS')
SECURE_SSL_REDIRECT = True
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
SECURE_HSTS_SECONDS = 31536000
SECURE_HSTS_INCLUDE_SUBDOMAINS = True
SECURE_HSTS_PRELOAD = True
# Logging
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
'file': {
'level': 'WARNING',
'class': 'logging.FileHandler',
'filename': '/var/log/django/django.log',
},
},
'loggers': {
'django': {
'handlers': ['file'],
'level': 'WARNING',
'propagate': True,
},
},
}
Model Design Patterns
Model Best Practices
from django.db import models
from django.contrib.auth.models import AbstractUser
from django.core.validators import MinValueValidator, MaxValueValidator
class User(AbstractUser):
"""Custom user model extending AbstractUser."""
email = models.EmailField(unique=True)
phone = models.CharField(max_length=20, blank=True)
birth_date = models.DateField(null=True, blank=True)
USERNAME_FIELD = 'email'
REQUIRED_FIELDS = ['username']
class Meta:
db_table = 'users'
verbose_name = 'user'
verbose_name_plural = 'users'
ordering = ['-date_joined']
def __str__(self):
return self.email
def get_full_name(self):
return f"{self.first_name} {self.last_name}".strip()
class Product(models.Model):
"""Product model with proper field configuration."""
name = models.CharField(max_length=200)
slug = models.SlugField(unique=True, max_length=250)
description = models.TextField(blank=True)
price = models.DecimalField(
max_digits=10,
decimal_places=2,
validators=[MinValueValidator(0)]
)
stock = models.PositiveIntegerField(default=0)
is_active = models.BooleanField(default=True)
category = models.ForeignKey(
'Category',
on_delete=models.CASCADE,
related_name='products'
)
tags = models.ManyToManyField('Tag', blank=True, related_name='products')
created_at = models.DateTimeField(auto_now_add=True)
updated_at = models.DateTimeField(auto_now=True)
class Meta:
db_table = 'products'
ordering = ['-created_at']
indexes = [
models.Index(fields=['slug']),
models.Index(fields=['-created_at']),
models.Index(fields=['category', 'is_active']),
]
constraints = [
models.CheckConstraint(
check=models.Q(price__gte=0),
name='price_non_negative'
)
]
def __str__(self):
return self.name
def save(self, *args, **kwargs):
if not self.slug:
self.slug = slugify(self.name)
super().save(*args, **kwargs)
QuerySet Best Practices
from django.db import models
class ProductQuerySet(models.QuerySet):
"""Custom QuerySet for Product model."""
def active(self):
"""Return only active products."""
return self.filter(is_active=True)
def with_category(self):
"""Select related category to avoid N+1 queries."""
return self.select_related('category')
def with_tags(self):
"""Prefetch tags for many-to-many relationship."""
return self.prefetch_related('tags')
def in_stock(self):
"""Return products with stock > 0."""
return self.filter(stock__gt=0)
def search(self, query):
"""Search products by name or description."""
return self.filter(
models.Q(name__icontains=query) |
models.Q(description__icontains=query)
)
class Product(models.Model):
# ... fields ...
objects = ProductQuerySet.as_manager() # Use custom QuerySet
# Usage
Product.objects.active().with_category().in_stock()
Manager Methods
class ProductManager(models.Manager):
"""Custom manager for complex queries."""
def get_or_none(self, **kwargs):
"""Return object or None instead of DoesNotExist."""
try:
return self.get(**kwargs)
except self.model.DoesNotExist:
return None
def create_with_tags(self, name, price, tag_names):
"""Create product with associated tags."""
product = self.create(name=name, price=price)
tags = [Tag.objects.get_or_create(name=name)[0] for name in tag_names]
product.tags.set(tags)
return product
def bulk_update_stock(self, product_ids, quantity):
"""Bulk update stock for multiple products."""
return self.filter(id__in=product_ids).update(stock=quantity)
# In model
class Product(models.Model):
# ... fields ...
custom = ProductManager()
Django REST Framework Patterns
Serializer Patterns
from rest_framework import serializers
from django.contrib.auth.password_validation import validate_password
from .models import Product, User
class ProductSerializer(serializers.ModelSerializer):
"""Serializer for Product model."""
category_name = serializers.CharField(source='category.name', read_only=True)
average_rating = serializers.FloatField(read_only=True)
discount_price = serializers.SerializerMethodField()
class Meta:
model = Product
fields = [
'id', 'name', 'slug', 'description', 'price',
'discount_price', 'stock', 'category_name',
'average_rating', 'created_at'
]
read_only_fields = ['id', 'slug', 'created_at']
def get_discount_price(self, obj):
"""Calculate discount price if applicable."""
if hasattr(obj, 'discount') and obj.discount:
return obj.price * (1 - obj.discount.percent / 100)
return obj.price
def validate_price(self, value):
"""Ensure price is non-negative."""
if value < 0:
raise serializers.ValidationError("Price cannot be negative.")
return value
class ProductCreateSerializer(serializers.ModelSerializer):
"""Serializer for creating products."""
class Meta:
model = Product
fields = ['name', 'description', 'price', 'stock', 'category']
def validate(self, data):
"""Custom validation for multiple fields."""
if data['price'] > 10000 and data['stock'] > 100:
raise serializers.ValidationError(
"Cannot have high-value products with large stock."
)
return data
class UserRegistrationSerializer(serializers.ModelSerializer):
"""Serializer for user registration."""
password = serializers.CharField(
write_only=True,
required=True,
validators=[validate_password],
style={'input_type': 'password'}
)
password_confirm = serializers.CharField(write_only=True, style={'input_type': 'password'})
class Meta:
model = User
fields = ['email', 'username', 'password', 'password_confirm']
def validate(self, data):
"""Validate passwords match."""
if data['password'] != data['password_confirm']:
raise serializers.ValidationError({
"password_confirm": "Password fields didn't match."
})
return data
def create(self, validated_data):
"""Create user with hashed password."""
validated_data.pop('password_confirm')
password = validated_data.pop('password')
user = User.objects.create(**validated_data)
user.set_password(password)
user.save()
return user
ViewSet Patterns
from rest_framework import viewsets, status, filters
from rest_framework.decorators import action
from rest_framework.response import Response
from rest_framework.permissions import IsAuthenticated, IsAdminUser
from django_filters.rest_framework import DjangoFilterBackend
from .models import Product
from .serializers import ProductSerializer, ProductCreateSerializer
from .permissions import IsOwnerOrReadOnly
from .filters import ProductFilter
from .services import ProductService
class ProductViewSet(viewsets.ModelViewSet):
"""ViewSet for Product model."""
queryset = Product.objects.select_related('category').prefetch_related('tags')
permission_classes = [IsAuthenticated, IsOwnerOrReadOnly]
filter_backends = [DjangoFilterBackend, filters.SearchFilter, filters.OrderingFilter]
filterset_class = ProductFilter
search_fields = ['name', 'description']
ordering_fields = ['price', 'created_at', 'name']
ordering = ['-created_at']
def get_serializer_class(self):
"""Return appropriate serializer based on action."""
if self.action == 'create':
return ProductCreateSerializer
return ProductSerializer
def perform_create(self, serializer):
"""Save with user context."""
serializer.save(created_by=self.request.user)
@action(detail=False, methods=['get'])
def featured(self, request):
"""Return featured products."""
featured = self.queryset.filter(is_featured=True)[:10]
serializer = self.get_serializer(featured, many=True)
return Response(serializer.data)
@action(detail=True, methods=['post'])
def purchase(self, request, pk=None):
"""Purchase a product."""
product = self.get_object()
service = ProductService()
result = service.purchase(product, request.user)
return Response(result, status=status.HTTP_201_CREATED)
@action(detail=False, methods=['get'], permission_classes=[IsAuthenticated])
def my_products(self, request):
"""Return products created by current user."""
products = self.queryset.filter(created_by=request.user)
page = self.paginate_queryset(products)
serializer = self.get_serializer(page, many=True)
return self.get_paginated_response(serializer.data)
Custom Actions
from rest_framework.decorators import api_view, permission_classes
from rest_framework.permissions import IsAuthenticated
from rest_framework.response import Response
@api_view(['POST'])
@permission_classes([IsAuthenticated])
def add_to_cart(request):
"""Add product to user cart."""
product_id = request.data.get('product_id')
quantity = request.data.get('quantity', 1)
try:
product = Product.objects.get(id=product_id)
except Product.DoesNotExist:
return Response(
{'error': 'Product not found'},
status=status.HTTP_404_NOT_FOUND
)
cart, _ = Cart.objects.get_or_create(user=request.user)
CartItem.objects.create(
cart=cart,
product=product,
quantity=quantity
)
return Response({'message': 'Added to cart'}, status=status.HTTP_201_CREATED)
Service Layer Pattern
# apps/orders/services.py
from typing import Optional
from django.db import transaction
from .models import Order, OrderItem
class OrderService:
"""Service layer for order-related business logic."""
@staticmethod
@transaction.atomic
def create_order(user, cart: Cart) -> Order:
"""Create order from cart."""
order = Order.objects.create(
user=user,
total_price=cart.total_price
)
for item in cart.items.all():
OrderItem.objects.create(
order=order,
product=item.product,
quantity=item.quantity,
price=item.product.price
)
# Clear cart
cart.items.all().delete()
return order
@staticmethod
def process_payment(order: Order, payment_data: dict) -> bool:
"""Process payment for order."""
# Integration with payment gateway
payment = PaymentGateway.charge(
amount=order.total_price,
token=payment_data['token']
)
if payment.success:
order.status = Order.Status.PAID
order.save()
# Send confirmation email
OrderService.send_confirmation_email(order)
return True
return False
@staticmethod
def send_confirmation_email(order: Order):
"""Send order confirmation email."""
# Email sending logic
pass
Caching Strategies
View-Level Caching
from django.views.decorators.cache import cache_page
from django.utils.decorators import method_decorator
@method_decorator(cache_page(60 * 15), name='dispatch') # 15 minutes
class ProductListView(generic.ListView):
model = Product
template_name = 'products/list.html'
context_object_name = 'products'
Template Fragment Caching
{% load cache %}
{% cache 500 sidebar %}
... expensive sidebar content ...
{% endcache %}
Low-Level Caching
from django.core.cache import cache
def get_featured_products():
"""Get featured products with caching."""
cache_key = 'featured_products'
products = cache.get(cache_key)
if products is None:
products = list(Product.objects.filter(is_featured=True))
cache.set(cache_key, products, timeout=60 * 15) # 15 minutes
return products
QuerySet Caching
from django.core.cache import cache
def get_popular_categories():
cache_key = 'popular_categories'
categories = cache.get(cache_key)
if categories is None:
categories = list(Category.objects.annotate(
product_count=Count('products')
).filter(product_count__gt=10).order_by('-product_count')[:20])
cache.set(cache_key, categories, timeout=60 * 60) # 1 hour
return categories
Signals
Signal Patterns
# apps/users/signals.py
from django.db.models.signals import post_save
from django.dispatch import receiver
from django.contrib.auth import get_user_model
from .models import Profile
User = get_user_model()
@receiver(post_save, sender=User)
def create_user_profile(sender, instance, created, **kwargs):
"""Create profile when user is created."""
if created:
Profile.objects.create(user=instance)
@receiver(post_save, sender=User)
def save_user_profile(sender, instance, **kwargs):
"""Save profile when user is saved."""
instance.profile.save()
# apps/users/apps.py
from django.apps import AppConfig
class UsersConfig(AppConfig):
default_auto_field = 'django.db.models.BigAutoField'
name = 'apps.users'
def ready(self):
"""Import signals when app is ready."""
import apps.users.signals
Middleware
Custom Middleware
# middleware/active_user_middleware.py
import time
from django.utils.deprecation import MiddlewareMixin
class ActiveUserMiddleware(MiddlewareMixin):
"""Middleware to track active users."""
def process_request(self, request):
"""Process incoming request."""
if request.user.is_authenticated:
# Update last active time
request.user.last_active = timezone.now()
request.user.save(update_fields=['last_active'])
class RequestLoggingMiddleware(MiddlewareMixin):
"""Middleware for logging requests."""
def process_request(self, request):
"""Log request start time."""
request.start_time = time.time()
def process_response(self, request, response):
"""Log request duration."""
if hasattr(request, 'start_time'):
duration = time.time() - request.start_time
logger.info(f'{request.method} {request.path} - {response.status_code} - {duration:.3f}s')
return response
Performance Optimization
N+1 Query Prevention
# Bad - N+1 queries
products = Product.objects.all()
for product in products:
print(product.category.name) # Separate query for each product
# Good - Single query with select_related
products = Product.objects.select_related('category').all()
for product in products:
print(product.category.name)
# Good - Prefetch for many-to-many
products = Product.objects.prefetch_related('tags').all()
for product in products:
for tag in product.tags.all():
print(tag.name)
Database Indexing
class Product(models.Model):
name = models.CharField(max_length=200, db_index=True)
slug = models.SlugField(unique=True)
category = models.ForeignKey('Category', on_delete=models.CASCADE)
created_at = models.DateTimeField(auto_now_add=True)
class Meta:
indexes = [
models.Index(fields=['name']),
models.Index(fields=['-created_at']),
models.Index(fields=['category', 'created_at']),
]
Bulk Operations
# Bulk create
Product.objects.bulk_create([
Product(name=f'Product {i}', price=10.00)
for i in range(1000)
])
# Bulk update
products = Product.objects.all()[:100]
for product in products:
product.is_active = True
Product.objects.bulk_update(products, ['is_active'])
# Bulk delete
Product.objects.filter(stock=0).delete()
Quick Reference
| Pattern | Description |
|---|---|
| Split settings | Separate dev/prod/test settings |
| Custom QuerySet | Reusable query methods |
| Service Layer | Business logic separation |
| ViewSet | REST API endpoints |
| Serializer validation | Request/response transformation |
| select_related | Foreign key optimization |
| prefetch_related | Many-to-many optimization |
| Cache first | Cache expensive operations |
| Signals | Event-driven actions |
| Middleware | Request/response processing |
Remember: Django provides many shortcuts, but for production applications, structure and organization matter more than concise code. Build for maintainability.
Django Security Best Practices
Comprehensive security guidelines for Django applications to protect against common vulnerabilities.
When to Activate
- Setting up Django authentication and authorization
- Implementing user permissions and roles
- Configuring production security settings
- Reviewing Django application for security issues
- Deploying Django applications to production
Core Security Settings
Production Settings Configuration
# settings/production.py
import os
DEBUG = False # CRITICAL: Never use True in production
ALLOWED_HOSTS = os.environ.get('ALLOWED_HOSTS', '').split(',')
# Security headers
SECURE_SSL_REDIRECT = True
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
SECURE_HSTS_SECONDS = 31536000 # 1 year
SECURE_HSTS_INCLUDE_SUBDOMAINS = True
SECURE_HSTS_PRELOAD = True
SECURE_CONTENT_TYPE_NOSNIFF = True
X_FRAME_OPTIONS = 'DENY'
# HTTPS and Cookies
SESSION_COOKIE_HTTPONLY = True
SESSION_COOKIE_SAMESITE = 'Lax'
CSRF_COOKIE_SAMESITE = 'Lax'
# Secret key (must be set via environment variable)
SECRET_KEY = os.environ.get('DJANGO_SECRET_KEY')
if not SECRET_KEY:
raise ImproperlyConfigured('DJANGO_SECRET_KEY environment variable is required')
# Password validation
AUTH_PASSWORD_VALIDATORS = [
{
'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
'OPTIONS': {
'min_length': 12,
}
},
{
'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
},
]
Authentication
Custom User Model
# apps/users/models.py
from django.contrib.auth.models import AbstractUser
from django.db import models
class User(AbstractUser):
"""Custom user model for better security."""
email = models.EmailField(unique=True)
phone = models.CharField(max_length=20, blank=True)
USERNAME_FIELD = 'email' # Use email as username
REQUIRED_FIELDS = ['username']
class Meta:
db_table = 'users'
verbose_name = 'User'
verbose_name_plural = 'Users'
def __str__(self):
return self.email
# settings/base.py
AUTH_USER_MODEL = 'users.User'
Password Hashing
# Django uses PBKDF2 by default. For stronger security:
PASSWORD_HASHERS = [
'django.contrib.auth.hashers.Argon2PasswordHasher',
'django.contrib.auth.hashers.PBKDF2PasswordHasher',
'django.contrib.auth.hashers.PBKDF2SHA1PasswordHasher',
'django.contrib.auth.hashers.BCryptSHA256PasswordHasher',
]
Session Management
# Session configuration
SESSION_ENGINE = 'django.contrib.sessions.backends.cache' # Or 'db'
SESSION_CACHE_ALIAS = 'default'
SESSION_COOKIE_AGE = 3600 * 24 * 7 # 1 week
SESSION_SAVE_EVERY_REQUEST = False
SESSION_EXPIRE_AT_BROWSER_CLOSE = False # Better UX, but less secure
Authorization
Permissions
# models.py
from django.db import models
from django.contrib.auth.models import Permission
class Post(models.Model):
title = models.CharField(max_length=200)
content = models.TextField()
author = models.ForeignKey(User, on_delete=models.CASCADE)
class Meta:
permissions = [
('can_publish', 'Can publish posts'),
('can_edit_others', 'Can edit posts of others'),
]
def user_can_edit(self, user):
"""Check if user can edit this post."""
return self.author == user or user.has_perm('app.can_edit_others')
# views.py
from django.contrib.auth.mixins import LoginRequiredMixin, PermissionRequiredMixin
from django.views.generic import UpdateView
class PostUpdateView(LoginRequiredMixin, PermissionRequiredMixin, UpdateView):
model = Post
permission_required = 'app.can_edit_others'
raise_exception = True # Return 403 instead of redirect
def get_queryset(self):
"""Only allow users to edit their own posts."""
return Post.objects.filter(author=self.request.user)
Custom Permissions
# permissions.py
from rest_framework import permissions
class IsOwnerOrReadOnly(permissions.BasePermission):
"""Allow only owners to edit objects."""
def has_object_permission(self, request, view, obj):
# Read permissions allowed for any request
if request.method in permissions.SAFE_METHODS:
return True
# Write permissions only for owner
return obj.author == request.user
class IsAdminOrReadOnly(permissions.BasePermission):
"""Allow admins to do anything, others read-only."""
def has_permission(self, request, view):
if request.method in permissions.SAFE_METHODS:
return True
return request.user and request.user.is_staff
class IsVerifiedUser(permissions.BasePermission):
"""Allow only verified users."""
def has_permission(self, request, view):
return request.user and request.user.is_authenticated and request.user.is_verified
Role-Based Access Control (RBAC)
# models.py
from django.contrib.auth.models import AbstractUser, Group
class User(AbstractUser):
ROLE_CHOICES = [
('admin', 'Administrator'),
('moderator', 'Moderator'),
('user', 'Regular User'),
]
role = models.CharField(max_length=20, choices=ROLE_CHOICES, default='user')
def is_admin(self):
return self.role == 'admin' or self.is_superuser
def is_moderator(self):
return self.role in ['admin', 'moderator']
# Mixins
class AdminRequiredMixin:
"""Mixin to require admin role."""
def dispatch(self, request, *args, **kwargs):
if not request.user.is_authenticated or not request.user.is_admin():
from django.core.exceptions import PermissionDenied
raise PermissionDenied
return super().dispatch(request, *args, **kwargs)
SQL Injection Prevention
Django ORM Protection
# GOOD: Django ORM automatically escapes parameters
def get_user(username):
return User.objects.get(username=username) # Safe
# GOOD: Using parameters with raw()
def search_users(query):
return User.objects.raw('SELECT * FROM users WHERE username = %s', [query])
# BAD: Never directly interpolate user input
def get_user_bad(username):
return User.objects.raw(f'SELECT * FROM users WHERE username = {username}') # VULNERABLE!
# GOOD: Using filter with proper escaping
def get_users_by_email(email):
return User.objects.filter(email__iexact=email) # Safe
# GOOD: Using Q objects for complex queries
from django.db.models import Q
def search_users_complex(query):
return User.objects.filter(
Q(username__icontains=query) |
Q(email__icontains=query)
) # Safe
Extra Security with raw()
# If you must use raw SQL, always use parameters
User.objects.raw(
'SELECT * FROM users WHERE email = %s AND status = %s',
[user_input_email, status]
)
XSS Prevention
Template Escaping
{# Django auto-escapes variables by default - SAFE #}
{{ user_input }} {# Escaped HTML #}
{# Explicitly mark safe only for trusted content #}
{{ trusted_html|safe }} {# Not escaped #}
{# Use template filters for safe HTML #}
{{ user_input|escape }} {# Same as default #}
{{ user_input|striptags }} {# Remove all HTML tags #}
{# JavaScript escaping #}
<script>
var username = {{ username|escapejs }};
</script>
Safe String Handling
from django.utils.safestring import mark_safe
from django.utils.html import escape
# BAD: Never mark user input as safe without escaping
def render_bad(user_input):
return mark_safe(user_input) # VULNERABLE!
# GOOD: Escape first, then mark safe
def render_good(user_input):
return mark_safe(escape(user_input))
# GOOD: Use format_html for HTML with variables
from django.utils.html import format_html
def greet_user(username):
return format_html('<span class="user">{}</span>', escape(username))
HTTP Headers
# settings.py
SECURE_CONTENT_TYPE_NOSNIFF = True # Prevent MIME sniffing
X_FRAME_OPTIONS = 'DENY' # Prevent clickjacking
# Custom middleware
from django.conf import settings
class SecurityHeaderMiddleware:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
response = self.get_response(request)
response['X-Content-Type-Options'] = 'nosniff'
response['X-Frame-Options'] = 'DENY'
response['X-XSS-Protection'] = '1; mode=block'
response['Content-Security-Policy'] = "default-src 'self'"
return response
CSRF Protection
Default CSRF Protection
# settings.py - CSRF is enabled by default
CSRF_COOKIE_SECURE = True # Only send over HTTPS
CSRF_COOKIE_HTTPONLY = False # False so AJAX can read csrf token from document.cookie; SESSION_COOKIE_HTTPONLY remains True
CSRF_COOKIE_SAMESITE = 'Lax' # Prevent CSRF in some cases
CSRF_TRUSTED_ORIGINS = ['https://example.com'] # Trusted domains
# Template usage
<form method="post">
{% csrf_token %}
{{ form.as_p }}
<button type="submit">Submit</button>
</form>
# AJAX requests
function getCookie(name) {
let cookieValue = null;
if (document.cookie && document.cookie !== '') {
const cookies = document.cookie.split(';');
for (let i = 0; i < cookies.length; i++) {
const cookie = cookies[i].trim();
if (cookie.substring(0, name.length + 1) === (name + '=')) {
cookieValue = decodeURIComponent(cookie.substring(name.length + 1));
break;
}
}
}
return cookieValue;
}
fetch('/api/endpoint/', {
method: 'POST',
headers: {
'X-CSRFToken': getCookie('csrftoken'),
'Content-Type': 'application/json',
},
body: JSON.stringify(data)
});
Exempting Views (Use Carefully)
from django.views.decorators.csrf import csrf_exempt
@csrf_exempt # Only use when absolutely necessary!
def webhook_view(request):
# Webhook from external service
pass
File Upload Security
File Validation
import os
import magic # pip install python-magic
from django.core.exceptions import ValidationError
ALLOWED_MIMES = {
'image/jpeg', 'image/png', 'image/gif', 'application/pdf',
}
MIME_TO_EXTENSIONS = {
'image/jpeg': {'.jpg', '.jpeg'},
'image/png': {'.png'},
'image/gif': {'.gif'},
'application/pdf': {'.pdf'},
}
def validate_file_type(value):
"""Validate file type using magic bytes and cross-check extension."""
mime = magic.from_buffer(value.read(2048), mime=True)
value.seek(0)
if mime not in ALLOWED_MIMES:
raise ValidationError('Unsupported file type.')
ext = os.path.splitext(value.name)[1].lower()
if ext not in MIME_TO_EXTENSIONS.get(mime, set()):
raise ValidationError('File extension does not match file content.')
def validate_file_size(value):
"""Validate file size (max 5MB)."""
if value.size > 5 * 1024 * 1024:
raise ValidationError('File too large. Max size is 5MB.')
# models.py
class Document(models.Model):
file = models.FileField(
upload_to='documents/',
validators=[validate_file_type, validate_file_size]
)
For environments where installing libmagic is difficult (e.g., minimal containers),
use the pure-Python filetype package as an alternative:
import os
from django.core.exceptions import ValidationError
import filetype # pip install filetype
ALLOWED_MIMES = {
'image/jpeg', 'image/png', 'image/gif', 'application/pdf',
}
MIME_TO_EXTENSIONS = {
'image/jpeg': {'.jpg', '.jpeg'},
'image/png': {'.png'},
'image/gif': {'.gif'},
'application/pdf': {'.pdf'},
}
def validate_file_type(value):
"""Validate file type using magic bytes."""
kind = filetype.guess(value.read(2048))
value.seek(0)
if kind is None or kind.mime not in ALLOWED_MIMES:
raise ValidationError('Unsupported file type.')
ext = os.path.splitext(value.name)[1].lower()
if ext not in MIME_TO_EXTENSIONS.get(kind.mime, set()):
raise ValidationError('File extension does not match file content.')
Secure File Storage
# settings.py
MEDIA_ROOT = '/var/www/media/'
MEDIA_URL = '/media/'
# Use a separate domain for media in production
MEDIA_DOMAIN = 'https://media.example.com'
# Don't serve user uploads directly
# Use whitenoise or a CDN for static files
# Use a separate server or S3 for media files
API Security
Rate Limiting
# settings.py
REST_FRAMEWORK = {
'DEFAULT_THROTTLE_CLASSES': [
'rest_framework.throttling.AnonRateThrottle',
'rest_framework.throttling.UserRateThrottle'
],
'DEFAULT_THROTTLE_RATES': {
'anon': '100/day',
'user': '1000/day',
'upload': '10/hour',
}
}
# Custom throttle
from rest_framework.throttling import UserRateThrottle
class BurstRateThrottle(UserRateThrottle):
scope = 'burst'
rate = '60/min'
class SustainedRateThrottle(UserRateThrottle):
scope = 'sustained'
rate = '1000/day'
Authentication for APIs
# settings.py
REST_FRAMEWORK = {
'DEFAULT_AUTHENTICATION_CLASSES': [
'rest_framework.authentication.TokenAuthentication',
'rest_framework.authentication.SessionAuthentication',
'rest_framework_simplejwt.authentication.JWTAuthentication',
],
'DEFAULT_PERMISSION_CLASSES': [
'rest_framework.permissions.IsAuthenticated',
],
}
# views.py
from rest_framework.decorators import api_view, permission_classes
from rest_framework.permissions import IsAuthenticated
@api_view(['GET', 'POST'])
@permission_classes([IsAuthenticated])
def protected_view(request):
return Response({'message': 'You are authenticated'})
Security Headers
Content Security Policy
# settings.py
CSP_DEFAULT_SRC = "'self'"
CSP_SCRIPT_SRC = "'self' https://cdn.example.com"
CSP_STYLE_SRC = "'self' 'unsafe-inline'"
CSP_IMG_SRC = "'self' data: https:"
CSP_CONNECT_SRC = "'self' https://api.example.com"
# Middleware
class CSPMiddleware:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
response = self.get_response(request)
response['Content-Security-Policy'] = (
f"default-src {CSP_DEFAULT_SRC}; "
f"script-src {CSP_SCRIPT_SRC}; "
f"style-src {CSP_STYLE_SRC}; "
f"img-src {CSP_IMG_SRC}; "
f"connect-src {CSP_CONNECT_SRC}"
)
return response
Environment Variables
Managing Secrets
# Use python-decouple or django-environ
import environ
env = environ.Env(
# set casting, default value
DEBUG=(bool, False)
)
# reading .env file
environ.Env.read_env()
SECRET_KEY = env('DJANGO_SECRET_KEY')
DATABASE_URL = env('DATABASE_URL')
ALLOWED_HOSTS = env.list('ALLOWED_HOSTS')
# .env file (never commit this)
DEBUG=False
SECRET_KEY=your-secret-key-here
DATABASE_URL=postgresql://user:password@localhost:5432/dbname
ALLOWED_HOSTS=example.com,www.example.com
Logging Security Events
# settings.py
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
'file': {
'level': 'WARNING',
'class': 'logging.FileHandler',
'filename': '/var/log/django/security.log',
},
'console': {
'level': 'INFO',
'class': 'logging.StreamHandler',
},
},
'loggers': {
'django.security': {
'handlers': ['file', 'console'],
'level': 'WARNING',
'propagate': True,
},
'django.request': {
'handlers': ['file'],
'level': 'ERROR',
'propagate': False,
},
},
}
Quick Security Checklist
| Check | Description |
|---|---|
DEBUG = False | Never run with DEBUG in production |
| HTTPS only | Force SSL, secure cookies |
| Strong secrets | Use environment variables for SECRET_KEY |
| Password validation | Enable all password validators |
| CSRF protection | Enabled by default, don't disable |
| XSS prevention | Django auto-escapes, don't use |safe with user input |
| SQL injection | Use ORM, never concatenate strings in queries |
| File uploads | Validate file type and size |
| Rate limiting | Throttle API endpoints |
| Security headers | CSP, X-Frame-Options, HSTS |
| Logging | Log security events |
| Updates | Keep Django and dependencies updated |
Remember: Security is a process, not a product. Regularly review and update your security practices.
Django Testing with TDD
Test-driven development for Django applications using pytest, factory_boy, and Django REST Framework.
When to Activate
- Writing new Django applications
- Implementing Django REST Framework APIs
- Testing Django models, views, and serializers
- Setting up testing infrastructure for Django projects
TDD Workflow for Django
Red-Green-Refactor Cycle
# Step 1: RED - Write failing test
def test_user_creation():
user = User.objects.create_user(email='test@example.com', password='testpass123')
assert user.email == 'test@example.com'
assert user.check_password('testpass123')
assert not user.is_staff
# Step 2: GREEN - Make test pass
# Create User model or factory
# Step 3: REFACTOR - Improve while keeping tests green
Setup
pytest Configuration
# pytest.ini
[pytest]
DJANGO_SETTINGS_MODULE = config.settings.test
testpaths = tests
python_files = test_*.py
python_classes = Test*
python_functions = test_*
addopts =
--reuse-db
--nomigrations
--cov=apps
--cov-report=html
--cov-report=term-missing
--strict-markers
markers =
slow: marks tests as slow
integration: marks tests as integration tests
Test Settings
# config/settings/test.py
from .base import *
DEBUG = True
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': ':memory:',
}
}
# Disable migrations for speed
class DisableMigrations:
def __contains__(self, item):
return True
def __getitem__(self, item):
return None
MIGRATION_MODULES = DisableMigrations()
# Faster password hashing
PASSWORD_HASHERS = [
'django.contrib.auth.hashers.MD5PasswordHasher',
]
# Email backend
EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'
# Celery always eager
CELERY_TASK_ALWAYS_EAGER = True
CELERY_TASK_EAGER_PROPAGATES = True
conftest.py
# tests/conftest.py
import pytest
from django.utils import timezone
from django.contrib.auth import get_user_model
User = get_user_model()
@pytest.fixture(autouse=True)
def timezone_settings(settings):
"""Ensure consistent timezone."""
settings.TIME_ZONE = 'UTC'
@pytest.fixture
def user(db):
"""Create a test user."""
return User.objects.create_user(
email='test@example.com',
password='testpass123',
username='testuser'
)
@pytest.fixture
def admin_user(db):
"""Create an admin user."""
return User.objects.create_superuser(
email='admin@example.com',
password='adminpass123',
username='admin'
)
@pytest.fixture
def authenticated_client(client, user):
"""Return authenticated client."""
client.force_login(user)
return client
@pytest.fixture
def api_client():
"""Return DRF API client."""
from rest_framework.test import APIClient
return APIClient()
@pytest.fixture
def authenticated_api_client(api_client, user):
"""Return authenticated API client."""
api_client.force_authenticate(user=user)
return api_client
Factory Boy
Factory Setup
# tests/factories.py
import factory
from factory import fuzzy
from datetime import datetime, timedelta
from django.contrib.auth import get_user_model
from apps.products.models import Product, Category
User = get_user_model()
class UserFactory(factory.django.DjangoModelFactory):
"""Factory for User model."""
class Meta:
model = User
email = factory.Sequence(lambda n: f"user{n}@example.com")
username = factory.Sequence(lambda n: f"user{n}")
password = factory.PostGenerationMethodCall('set_password', 'testpass123')
first_name = factory.Faker('first_name')
last_name = factory.Faker('last_name')
is_active = True
class CategoryFactory(factory.django.DjangoModelFactory):
"""Factory for Category model."""
class Meta:
model = Category
name = factory.Faker('word')
slug = factory.LazyAttribute(lambda obj: obj.name.lower())
description = factory.Faker('text')
class ProductFactory(factory.django.DjangoModelFactory):
"""Factory for Product model."""
class Meta:
model = Product
name = factory.Faker('sentence', nb_words=3)
slug = factory.LazyAttribute(lambda obj: obj.name.lower().replace(' ', '-'))
description = factory.Faker('text')
price = fuzzy.FuzzyDecimal(10.00, 1000.00, 2)
stock = fuzzy.FuzzyInteger(0, 100)
is_active = True
category = factory.SubFactory(CategoryFactory)
created_by = factory.SubFactory(UserFactory)
@factory.post_generation
def tags(self, create, extracted, **kwargs):
"""Add tags to product."""
if not create:
return
if extracted:
for tag in extracted:
self.tags.add(tag)
Using Factories
# tests/test_models.py
import pytest
from tests.factories import ProductFactory, UserFactory
def test_product_creation():
"""Test product creation using factory."""
product = ProductFactory(price=100.00, stock=50)
assert product.price == 100.00
assert product.stock == 50
assert product.is_active is True
def test_product_with_tags():
"""Test product with tags."""
tags = [TagFactory(name='electronics'), TagFactory(name='new')]
product = ProductFactory(tags=tags)
assert product.tags.count() == 2
def test_multiple_products():
"""Test creating multiple products."""
products = ProductFactory.create_batch(10)
assert len(products) == 10
Model Testing
Model Tests
# tests/test_models.py
import pytest
from django.core.exceptions import ValidationError
from tests.factories import UserFactory, ProductFactory
class TestUserModel:
"""Test User model."""
def test_create_user(self, db):
"""Test creating a regular user."""
user = UserFactory(email='test@example.com')
assert user.email == 'test@example.com'
assert user.check_password('testpass123')
assert not user.is_staff
assert not user.is_superuser
def test_create_superuser(self, db):
"""Test creating a superuser."""
user = UserFactory(
email='admin@example.com',
is_staff=True,
is_superuser=True
)
assert user.is_staff
assert user.is_superuser
def test_user_str(self, db):
"""Test user string representation."""
user = UserFactory(email='test@example.com')
assert str(user) == 'test@example.com'
class TestProductModel:
"""Test Product model."""
def test_product_creation(self, db):
"""Test creating a product."""
product = ProductFactory()
assert product.id is not None
assert product.is_active is True
assert product.created_at is not None
def test_product_slug_generation(self, db):
"""Test automatic slug generation."""
product = ProductFactory(name='Test Product')
assert product.slug == 'test-product'
def test_product_price_validation(self, db):
"""Test price cannot be negative."""
product = ProductFactory(price=-10)
with pytest.raises(ValidationError):
product.full_clean()
def test_product_manager_active(self, db):
"""Test active manager method."""
ProductFactory.create_batch(5, is_active=True)
ProductFactory.create_batch(3, is_active=False)
active_count = Product.objects.active().count()
assert active_count == 5
def test_product_stock_management(self, db):
"""Test stock management."""
product = ProductFactory(stock=10)
product.reduce_stock(5)
product.refresh_from_db()
assert product.stock == 5
with pytest.raises(ValueError):
product.reduce_stock(10) # Not enough stock
View Testing
Django View Testing
# tests/test_views.py
import pytest
from django.urls import reverse
from tests.factories import ProductFactory, UserFactory
class TestProductViews:
"""Test product views."""
def test_product_list(self, client, db):
"""Test product list view."""
ProductFactory.create_batch(10)
response = client.get(reverse('products:list'))
assert response.status_code == 200
assert len(response.context['products']) == 10
def test_product_detail(self, client, db):
"""Test product detail view."""
product = ProductFactory()
response = client.get(reverse('products:detail', kwargs={'slug': product.slug}))
assert response.status_code == 200
assert response.context['product'] == product
def test_product_create_requires_login(self, client, db):
"""Test product creation requires authentication."""
response = client.get(reverse('products:create'))
assert response.status_code == 302
assert response.url.startswith('/accounts/login/')
def test_product_create_authenticated(self, authenticated_client, db):
"""Test product creation as authenticated user."""
response = authenticated_client.get(reverse('products:create'))
assert response.status_code == 200
def test_product_create_post(self, authenticated_client, db, category):
"""Test creating a product via POST."""
data = {
'name': 'Test Product',
'description': 'A test product',
'price': '99.99',
'stock': 10,
'category': category.id,
}
response = authenticated_client.post(reverse('products:create'), data)
assert response.status_code == 302
assert Product.objects.filter(name='Test Product').exists()
DRF API Testing
Serializer Testing
# tests/test_serializers.py
import pytest
from rest_framework.exceptions import ValidationError
from apps.products.serializers import ProductSerializer
from tests.factories import ProductFactory
class TestProductSerializer:
"""Test ProductSerializer."""
def test_serialize_product(self, db):
"""Test serializing a product."""
product = ProductFactory()
serializer = ProductSerializer(product)
data = serializer.data
assert data['id'] == product.id
assert data['name'] == product.name
assert data['price'] == str(product.price)
def test_deserialize_product(self, db):
"""Test deserializing product data."""
data = {
'name': 'Test Product',
'description': 'Test description',
'price': '99.99',
'stock': 10,
'category': 1,
}
serializer = ProductSerializer(data=data)
assert serializer.is_valid()
product = serializer.save()
assert product.name == 'Test Product'
assert float(product.price) == 99.99
def test_price_validation(self, db):
"""Test price validation."""
data = {
'name': 'Test Product',
'price': '-10.00',
'stock': 10,
}
serializer = ProductSerializer(data=data)
assert not serializer.is_valid()
assert 'price' in serializer.errors
def test_stock_validation(self, db):
"""Test stock cannot be negative."""
data = {
'name': 'Test Product',
'price': '99.99',
'stock': -5,
}
serializer = ProductSerializer(data=data)
assert not serializer.is_valid()
assert 'stock' in serializer.errors
API ViewSet Testing
# tests/test_api.py
import pytest
from rest_framework.test import APIClient
from rest_framework import status
from django.urls import reverse
from tests.factories import ProductFactory, UserFactory
class TestProductAPI:
"""Test Product API endpoints."""
@pytest.fixture
def api_client(self):
"""Return API client."""
return APIClient()
def test_list_products(self, api_client, db):
"""Test listing products."""
ProductFactory.create_batch(10)
url = reverse('api:product-list')
response = api_client.get(url)
assert response.status_code == status.HTTP_200_OK
assert response.data['count'] == 10
def test_retrieve_product(self, api_client, db):
"""Test retrieving a product."""
product = ProductFactory()
url = reverse('api:product-detail', kwargs={'pk': product.id})
response = api_client.get(url)
assert response.status_code == status.HTTP_200_OK
assert response.data['id'] == product.id
def test_create_product_unauthorized(self, api_client, db):
"""Test creating product without authentication."""
url = reverse('api:product-list')
data = {'name': 'Test Product', 'price': '99.99'}
response = api_client.post(url, data)
assert response.status_code == status.HTTP_401_UNAUTHORIZED
def test_create_product_authorized(self, authenticated_api_client, db):
"""Test creating product as authenticated user."""
url = reverse('api:product-list')
data = {
'name': 'Test Product',
'description': 'Test',
'price': '99.99',
'stock': 10,
}
response = authenticated_api_client.post(url, data)
assert response.status_code == status.HTTP_201_CREATED
assert response.data['name'] == 'Test Product'
def test_update_product(self, authenticated_api_client, db):
"""Test updating a product."""
product = ProductFactory(created_by=authenticated_api_client.user)
url = reverse('api:product-detail', kwargs={'pk': product.id})
data = {'name': 'Updated Product'}
response = authenticated_api_client.patch(url, data)
assert response.status_code == status.HTTP_200_OK
assert response.data['name'] == 'Updated Product'
def test_delete_product(self, authenticated_api_client, db):
"""Test deleting a product."""
product = ProductFactory(created_by=authenticated_api_client.user)
url = reverse('api:product-detail', kwargs={'pk': product.id})
response = authenticated_api_client.delete(url)
assert response.status_code == status.HTTP_204_NO_CONTENT
def test_filter_products_by_price(self, api_client, db):
"""Test filtering products by price."""
ProductFactory(price=50)
ProductFactory(price=150)
url = reverse('api:product-list')
response = api_client.get(url, {'price_min': 100})
assert response.status_code == status.HTTP_200_OK
assert response.data['count'] == 1
def test_search_products(self, api_client, db):
"""Test searching products."""
ProductFactory(name='Apple iPhone')
ProductFactory(name='Samsung Galaxy')
url = reverse('api:product-list')
response = api_client.get(url, {'search': 'Apple'})
assert response.status_code == status.HTTP_200_OK
assert response.data['count'] == 1
Mocking and Patching
Mocking External Services
# tests/test_views.py
from unittest.mock import patch, Mock
import pytest
class TestPaymentView:
"""Test payment view with mocked payment gateway."""
@patch('apps.payments.services.stripe')
def test_successful_payment(self, mock_stripe, client, user, product):
"""Test successful payment with mocked Stripe."""
# Configure mock
mock_stripe.Charge.create.return_value = {
'id': 'ch_123',
'status': 'succeeded',
'amount': 9999,
}
client.force_login(user)
response = client.post(reverse('payments:process'), {
'product_id': product.id,
'token': 'tok_visa',
})
assert response.status_code == 302
mock_stripe.Charge.create.assert_called_once()
@patch('apps.payments.services.stripe')
def test_failed_payment(self, mock_stripe, client, user, product):
"""Test failed payment."""
mock_stripe.Charge.create.side_effect = Exception('Card declined')
client.force_login(user)
response = client.post(reverse('payments:process'), {
'product_id': product.id,
'token': 'tok_visa',
})
assert response.status_code == 302
assert 'error' in response.url
Mocking Email Sending
# tests/test_email.py
from django.core import mail
from django.test import override_settings
@override_settings(EMAIL_BACKEND='django.core.mail.backends.locmem.EmailBackend')
def test_order_confirmation_email(db, order):
"""Test order confirmation email."""
order.send_confirmation_email()
assert len(mail.outbox) == 1
assert order.user.email in mail.outbox[0].to
assert 'Order Confirmation' in mail.outbox[0].subject
Integration Testing
Full Flow Testing
# tests/test_integration.py
import pytest
from django.urls import reverse
from tests.factories import UserFactory, ProductFactory
class TestCheckoutFlow:
"""Test complete checkout flow."""
def test_guest_to_purchase_flow(self, client, db):
"""Test complete flow from guest to purchase."""
# Step 1: Register
response = client.post(reverse('users:register'), {
'email': 'test@example.com',
'password': 'testpass123',
'password_confirm': 'testpass123',
})
assert response.status_code == 302
# Step 2: Login
response = client.post(reverse('users:login'), {
'email': 'test@example.com',
'password': 'testpass123',
})
assert response.status_code == 302
# Step 3: Browse products
product = ProductFactory(price=100)
response = client.get(reverse('products:detail', kwargs={'slug': product.slug}))
assert response.status_code == 200
# Step 4: Add to cart
response = client.post(reverse('cart:add'), {
'product_id': product.id,
'quantity': 1,
})
assert response.status_code == 302
# Step 5: Checkout
response = client.get(reverse('checkout:review'))
assert response.status_code == 200
assert product.name in response.content.decode()
# Step 6: Complete purchase
with patch('apps.checkout.services.process_payment') as mock_payment:
mock_payment.return_value = True
response = client.post(reverse('checkout:complete'))
assert response.status_code == 302
assert Order.objects.filter(user__email='test@example.com').exists()
Testing Best Practices
DO
- Use factories: Instead of manual object creation
- One assertion per test: Keep tests focused
- Descriptive test names:
test_user_cannot_delete_others_post - Test edge cases: Empty inputs, None values, boundary conditions
- Mock external services: Don't depend on external APIs
- Use fixtures: Eliminate duplication
- Test permissions: Ensure authorization works
- Keep tests fast: Use
--reuse-dband--nomigrations
DON'T
- Don't test Django internals: Trust Django to work
- Don't test third-party code: Trust libraries to work
- Don't ignore failing tests: All tests must pass
- Don't make tests dependent: Tests should run in any order
- Don't over-mock: Mock only external dependencies
- Don't test private methods: Test public interface
- Don't use production database: Always use test database
Coverage
Coverage Configuration
# Run tests with coverage
pytest --cov=apps --cov-report=html --cov-report=term-missing
# Generate HTML report
open htmlcov/index.html
Coverage Goals
| Component | Target Coverage |
|---|---|
| Models | 90%+ |
| Serializers | 85%+ |
| Views | 80%+ |
| Services | 90%+ |
| Utilities | 80%+ |
| Overall | 80%+ |
Quick Reference
| Pattern | Usage |
|---|---|
@pytest.mark.django_db | Enable database access |
client | Django test client |
api_client | DRF API client |
factory.create_batch(n) | Create multiple objects |
patch('module.function') | Mock external dependencies |
override_settings | Temporarily change settings |
force_authenticate() | Bypass authentication in tests |
assertRedirects | Check for redirects |
assertTemplateUsed | Verify template usage |
mail.outbox | Check sent emails |
Remember: Tests are documentation. Good tests explain how your code should work. Keep them simple, readable, and maintainable.
Django Verification Loop
Run before PRs, after major changes, and pre-deploy to ensure Django application quality and security.
When to Activate
- Before opening a pull request for a Django project
- After major model changes, migration updates, or dependency upgrades
- Pre-deployment verification for staging or production
- Running full environment → lint → test → security → deploy readiness pipeline
- Validating migration safety and test coverage
Phase 1: Environment Check
# Verify Python version
python --version # Should match project requirements
# Check virtual environment
which python
pip list --outdated
# Verify environment variables
python -c "import os; import environ; print('DJANGO_SECRET_KEY set' if os.environ.get('DJANGO_SECRET_KEY') else 'MISSING: DJANGO_SECRET_KEY')"
If environment is misconfigured, stop and fix.
Phase 2: Code Quality & Formatting
# Type checking
mypy . --config-file pyproject.toml
# Linting with ruff
ruff check . --fix
# Formatting with black
black . --check
black . # Auto-fix
# Import sorting
isort . --check-only
isort . # Auto-fix
# Django-specific checks
python manage.py check --deploy
Common issues:
- Missing type hints on public functions
- PEP 8 formatting violations
- Unsorted imports
- Debug settings left in production configuration
Phase 3: Migrations
# Check for unapplied migrations
python manage.py showmigrations
# Create missing migrations
python manage.py makemigrations --check
# Dry-run migration application
python manage.py migrate --plan
# Apply migrations (test environment)
python manage.py migrate
# Check for migration conflicts
python manage.py makemigrations --merge # Only if conflicts exist
Report:
- Number of pending migrations
- Any migration conflicts
- Model changes without migrations
Phase 4: Tests + Coverage
# Run all tests with pytest
pytest --cov=apps --cov-report=html --cov-report=term-missing --reuse-db
# Run specific app tests
pytest apps/users/tests/
# Run with markers
pytest -m "not slow" # Skip slow tests
pytest -m integration # Only integration tests
# Coverage report
open htmlcov/index.html
Report:
- Total tests: X passed, Y failed, Z skipped
- Overall coverage: XX%
- Per-app coverage breakdown
Coverage targets:
| Component | Target |
|---|---|
| Models | 90%+ |
| Serializers | 85%+ |
| Views | 80%+ |
| Services | 90%+ |
| Overall | 80%+ |
Phase 5: Security Scan
# Dependency vulnerabilities
pip-audit
safety check --full-report
# Django security checks
python manage.py check --deploy
# Bandit security linter
bandit -r . -f json -o bandit-report.json
# Secret scanning (if gitleaks is installed)
gitleaks detect --source . --verbose
# Environment variable check
python -c "from django.core.exceptions import ImproperlyConfigured; from django.conf import settings; settings.DEBUG"
Report:
- Vulnerable dependencies found
- Security configuration issues
- Hardcoded secrets detected
- DEBUG mode status (should be False in production)
Phase 6: Django Management Commands
# Check for model issues
python manage.py check
# Collect static files
python manage.py collectstatic --noinput --clear
# Create superuser (if needed for tests)
echo "from apps.users.models import User; User.objects.create_superuser('admin@example.com', 'admin')" | python manage.py shell
# Database integrity
python manage.py check --database default
# Cache verification (if using Redis)
python -c "from django.core.cache import cache; cache.set('test', 'value', 10); print(cache.get('test'))"
Phase 7: Performance Checks
# Django Debug Toolbar output (check for N+1 queries)
# Run in dev mode with DEBUG=True and access a page
# Look for duplicate queries in SQL panel
# Query count analysis
django-admin debugsqlshell # If django-debug-sqlshell installed
# Check for missing indexes
python manage.py shell << EOF
from django.db import connection
with connection.cursor() as cursor:
cursor.execute("SELECT table_name, index_name FROM information_schema.statistics WHERE table_schema = 'public'")
print(cursor.fetchall())
EOF
Report:
- Number of queries per page (should be < 50 for typical pages)
- Missing database indexes
- Duplicate queries detected
Phase 8: Static Assets
# Check for npm dependencies (if using npm)
npm audit
npm audit fix
# Build static files (if using webpack/vite)
npm run build
# Verify static files
ls -la staticfiles/
python manage.py findstatic css/style.css
Phase 9: Configuration Review
# Run in Python shell to verify settings
python manage.py shell << EOF
from django.conf import settings
import os
# Critical checks
checks = {
'DEBUG is False': not settings.DEBUG,
'SECRET_KEY set': bool(settings.SECRET_KEY and len(settings.SECRET_KEY) > 30),
'ALLOWED_HOSTS set': len(settings.ALLOWED_HOSTS) > 0,
'HTTPS enabled': getattr(settings, 'SECURE_SSL_REDIRECT', False),
'HSTS enabled': getattr(settings, 'SECURE_HSTS_SECONDS', 0) > 0,
'Database configured': settings.DATABASES['default']['ENGINE'] != 'django.db.backends.sqlite3',
}
for check, result in checks.items():
status = '✓' if result else '✗'
print(f"{status} {check}")
EOF
Phase 10: Logging Configuration
# Test logging output
python manage.py shell << EOF
import logging
logger = logging.getLogger('django')
logger.warning('Test warning message')
logger.error('Test error message')
EOF
# Check log files (if configured)
tail -f /var/log/django/django.log
Phase 11: API Documentation (if DRF)
# Generate schema
python manage.py generateschema --format openapi-json > schema.json
# Validate schema
# Check if schema.json is valid JSON
python -c "import json; json.load(open('schema.json'))"
# Access Swagger UI (if using drf-yasg)
# Visit http://localhost:8000/swagger/ in browser
Phase 12: Diff Review
# Show diff statistics
git diff --stat
# Show actual changes
git diff
# Show changed files
git diff --name-only
# Check for common issues
git diff | grep -i "todo\|fixme\|hack\|xxx"
git diff | grep "print(" # Debug statements
git diff | grep "DEBUG = True" # Debug mode
git diff | grep "import pdb" # Debugger
Checklist:
- No debugging statements (print, pdb, breakpoint())
- No TODO/FIXME comments in critical code
- No hardcoded secrets or credentials
- Database migrations included for model changes
- Configuration changes documented
- Error handling present for external calls
- Transaction management where needed
Output Template
DJANGO VERIFICATION REPORT
==========================
Phase 1: Environment Check
✓ Python 3.11.5
✓ Virtual environment active
✓ All environment variables set
Phase 2: Code Quality
✓ mypy: No type errors
✗ ruff: 3 issues found (auto-fixed)
✓ black: No formatting issues
✓ isort: Imports properly sorted
✓ manage.py check: No issues
Phase 3: Migrations
✓ No unapplied migrations
✓ No migration conflicts
✓ All models have migrations
Phase 4: Tests + Coverage
Tests: 247 passed, 0 failed, 5 skipped
Coverage:
Overall: 87%
users: 92%
products: 89%
orders: 85%
payments: 91%
Phase 5: Security Scan
✗ pip-audit: 2 vulnerabilities found (fix required)
✓ safety check: No issues
✓ bandit: No security issues
✓ No secrets detected
✓ DEBUG = False
Phase 6: Django Commands
✓ collectstatic completed
✓ Database integrity OK
✓ Cache backend reachable
Phase 7: Performance
✓ No N+1 queries detected
✓ Database indexes configured
✓ Query count acceptable
Phase 8: Static Assets
✓ npm audit: No vulnerabilities
✓ Assets built successfully
✓ Static files collected
Phase 9: Configuration
✓ DEBUG = False
✓ SECRET_KEY configured
✓ ALLOWED_HOSTS set
✓ HTTPS enabled
✓ HSTS enabled
✓ Database configured
Phase 10: Logging
✓ Logging configured
✓ Log files writable
Phase 11: API Documentation
✓ Schema generated
✓ Swagger UI accessible
Phase 12: Diff Review
Files changed: 12
+450, -120 lines
✓ No debug statements
✓ No hardcoded secrets
✓ Migrations included
RECOMMENDATION: WARNING: Fix pip-audit vulnerabilities before deploying
NEXT STEPS:
1. Update vulnerable dependencies
2. Re-run security scan
3. Deploy to staging for final testing
Pre-Deployment Checklist
- All tests passing
- Coverage ≥ 80%
- No security vulnerabilities
- No unapplied migrations
- DEBUG = False in production settings
- SECRET_KEY properly configured
- ALLOWED_HOSTS set correctly
- Database backups enabled
- Static files collected and served
- Logging configured and working
- Error monitoring (Sentry, etc.) configured
- CDN configured (if applicable)
- Redis/cache backend configured
- Celery workers running (if applicable)
- HTTPS/SSL configured
- Environment variables documented
Continuous Integration
GitHub Actions Example
# .github/workflows/django-verification.yml
name: Django Verification
on: [push, pull_request]
jobs:
verify:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:14
env:
POSTGRES_PASSWORD: postgres
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.11'
- name: Cache pip
uses: actions/cache@v3
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
- name: Install dependencies
run: |
pip install -r requirements.txt
pip install ruff black mypy pytest pytest-django pytest-cov bandit safety pip-audit
- name: Code quality checks
run: |
ruff check .
black . --check
isort . --check-only
mypy .
- name: Security scan
run: |
bandit -r . -f json -o bandit-report.json
safety check --full-report
pip-audit
- name: Run tests
env:
DATABASE_URL: postgres://postgres:postgres@localhost:5432/test
DJANGO_SECRET_KEY: test-secret-key
run: |
pytest --cov=apps --cov-report=xml --cov-report=term-missing
- name: Upload coverage
uses: codecov/codecov-action@v3
Quick Reference
| Check | Command |
|---|---|
| Environment | python --version |
| Type checking | mypy . |
| Linting | ruff check . |
| Formatting | black . --check |
| Migrations | python manage.py makemigrations --check |
| Tests | pytest --cov=apps |
| Security | pip-audit && bandit -r . |
| Django check | python manage.py check --deploy |
| Collectstatic | python manage.py collectstatic --noinput |
| Diff stats | git diff --stat |
Remember: Automated verification catches common issues but doesn't replace manual code review and testing in staging environment.
FastAPI Patterns
Modern, production-grade FastAPI development: project layout, Pydantic v2 schemas, dependency injection, async patterns, auth, transactional service methods, and testing.
Project Structure
my_app/
|-- app/
| |-- main.py # App factory, lifespan, middleware
| |-- config.py # Settings via pydantic-settings
| |-- dependencies.py # Shared FastAPI dependencies
| |-- database.py # SQLAlchemy engine + session
| |-- routers/
| | `-- users.py
| |-- models/ # SQLAlchemy ORM models
| | `-- user.py
| |-- schemas/ # Pydantic request/response schemas
| | `-- user.py
| `-- services/ # Business logic layer
| `-- user_service.py
|-- tests/
| |-- conftest.py
| `-- test_users.py
|-- pyproject.toml
`-- .env
App Factory and Lifespan
# app/main.py
from contextlib import asynccontextmanager
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from app.config import settings
from app.database import engine, Base
from app.routers import users
@asynccontextmanager
async def lifespan(app: FastAPI):
# Automatically create tables on startup for ease of use in dev/demo environments.
# For strict production applications, manage schemas via Alembic migrations instead.
async with engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
yield
# Shutdown: close pooled resources.
await engine.dispose()
def create_app() -> FastAPI:
app = FastAPI(
title=settings.app_name,
version=settings.app_version,
lifespan=lifespan,
)
app.add_middleware(
CORSMiddleware,
allow_origins=settings.allowed_origins,
allow_credentials=settings.allow_credentials,
allow_methods=settings.allowed_methods,
allow_headers=settings.allowed_headers,
)
app.include_router(users.router, prefix="/users", tags=["users"])
return app
app = create_app()
Configuration with pydantic-settings
# app/config.py
from pydantic_settings import BaseSettings, SettingsConfigDict
class Settings(BaseSettings):
model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8")
app_name: str = "My App"
app_version: str = "0.1.0"
debug: bool = False
database_url: str
secret_key: str
algorithm: str = "HS256"
access_token_expire_minutes: int = 30
# Pydantic-settings v2 safely evaluates mutable list literals directly
allowed_origins: list[str] = ["http://localhost:3000"]
allowed_methods: list[str] = ["GET", "POST", "PATCH", "DELETE", "OPTIONS"]
allowed_headers: list[str] = ["Authorization", "Content-Type"]
allow_credentials: bool = True
settings = Settings()
Pydantic Schemas (v2)
# app/schemas/user.py
from datetime import datetime
from pydantic import BaseModel, EmailStr, Field, model_validator
class UserBase(BaseModel):
email: EmailStr
username: str = Field(min_length=3, max_length=50)
class UserCreate(UserBase):
password: str = Field(min_length=8)
password_confirm: str
@model_validator(mode="after")
def passwords_match(self) -> "UserCreate":
if self.password != self.password_confirm:
raise ValueError("Passwords do not match")
return self
class UserUpdate(BaseModel):
username: str | None = Field(default=None, min_length=3, max_length=50)
email: EmailStr | None = None
class UserResponse(UserBase):
id: int
is_active: bool
created_at: datetime
model_config = {"from_attributes": True}
class UserListResponse(BaseModel):
total: int
items: list[UserResponse]
Dependency Injection
# app/dependencies.py
from typing import Annotated, AsyncGenerator
from fastapi import Depends, HTTPException, status
from fastapi.security import OAuth2PasswordBearer
from jose import JWTError, jwt
from sqlalchemy.ext.asyncio import AsyncSession
from app.config import settings
from app.database import AsyncSessionLocal
from app.models.user import User
oauth2_scheme = OAuth2PasswordBearer(tokenUrl="/users/token")
async def get_db() -> AsyncGenerator[AsyncSession, None]:
async with AsyncSessionLocal() as session:
try:
yield session
except Exception:
await session.rollback()
raise
async def get_current_user(
token: Annotated[str, Depends(oauth2_scheme)],
db: Annotated[AsyncSession, Depends(get_db)],
) -> User:
credentials_exception = HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Could not validate credentials",
headers={"WWW-Authenticate": "Bearer"},
)
try:
payload = jwt.decode(token, settings.secret_key, algorithms=[settings.algorithm])
subject = payload.get("sub")
if subject is None:
raise credentials_exception
user_id = int(subject)
except (JWTError, TypeError, ValueError):
raise credentials_exception
user = await db.get(User, user_id)
if user is None:
raise credentials_exception
return user
async def get_current_active_user(
current_user: Annotated[User, Depends(get_current_user)],
) -> User:
if not current_user.is_active:
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Inactive user")
return current_user
DbDep = Annotated[AsyncSession, Depends(get_db)]
CurrentUserDep = Annotated[User, Depends(get_current_user)]
ActiveUserDep = Annotated[User, Depends(get_current_active_user)]
Router and Endpoint Design
# app/routers/users.py
from typing import Annotated
from fastapi import APIRouter, Depends, HTTPException, Query, status
from fastapi.security import OAuth2PasswordRequestForm
from app.dependencies import ActiveUserDep, DbDep
from app.schemas.user import UserCreate, UserResponse, UserUpdate, UserListResponse
from app.services.user_service import DuplicateUserError, UserService
router = APIRouter()
@router.post("/", response_model=UserResponse, status_code=status.HTTP_201_CREATED)
async def create_user(payload: UserCreate, db: DbDep) -> UserResponse:
service = UserService(db)
try:
return await service.create(payload)
except DuplicateUserError:
raise HTTPException(status_code=400, detail="Email already registered")
@router.get("/me", response_model=UserResponse)
async def get_me(current_user: ActiveUserDep) -> UserResponse:
return current_user
@router.get("/", response_model=UserListResponse)
async def list_users(
db: DbDep,
current_user: ActiveUserDep,
skip: Annotated[int, Query(ge=0)] = 0,
limit: Annotated[int, Query(ge=1, le=100)] = 20,
) -> UserListResponse:
service = UserService(db)
users, total = await service.list(skip=skip, limit=limit)
return UserListResponse(total=total, items=users)
@router.patch("/{user_id}", response_model=UserResponse)
async def update_user(
user_id: int,
payload: UserUpdate,
db: DbDep,
current_user: ActiveUserDep,
) -> UserResponse:
if current_user.id != user_id:
raise HTTPException(status_code=403, detail="Not authorized")
service = UserService(db)
try:
user = await service.update(user_id, payload)
except DuplicateUserError:
raise HTTPException(status_code=400, detail="Email already registered")
if user is None:
raise HTTPException(status_code=404, detail="User not found")
return user
@router.post("/token")
async def login(
form_data: Annotated[OAuth2PasswordRequestForm, Depends()],
db: DbDep,
) -> dict[str, str]:
service = UserService(db)
token = await service.authenticate(form_data.username, form_data.password)
if token is None:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Incorrect username or password",
headers={"WWW-Authenticate": "Bearer"},
)
return {"access_token": token, "token_type": "bearer"}
Service Layer
# app/services/user_service.py
from datetime import datetime, timedelta, timezone
from jose import jwt
from passlib.context import CryptContext
from sqlalchemy import func, select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.asyncio import AsyncSession
from app.config import settings
from app.models.user import User
from app.schemas.user import UserCreate, UserUpdate
pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto")
class DuplicateUserError(Exception):
"""Raised when a unique user field conflicts with an existing row."""
class UserService:
def __init__(self, db: AsyncSession) -> None:
self.db = db
async def get_by_email(self, email: str) -> User | None:
result = await self.db.execute(select(User).where(User.email == email))
return result.scalar_one_or_none()
async def create(self, payload: UserCreate) -> User:
user = User(
email=payload.email,
username=payload.username,
hashed_password=pwd_context.hash(payload.password),
)
self.db.add(user)
try:
# Rely on atomic DB constraints rather than race-prone application-level prechecks
await self.db.commit()
except IntegrityError as exc:
await self.db.rollback()
raise DuplicateUserError from exc
await self.db.refresh(user)
return user
async def list(self, skip: int = 0, limit: int = 20) -> tuple[list[User], int]:
total_result = await self.db.execute(select(func.count(User.id)))
total = total_result.scalar_one()
# Enforce explicit deterministic ordering to ensure reliable pagination
result = await self.db.execute(
select(User).order_by(User.id).offset(skip).limit(limit)
)
return list(result.scalars()), total
async def update(self, user_id: int, payload: UserUpdate) -> User | None:
user = await self.db.get(User, user_id)
if user is None:
return None
for field, value in payload.model_dump(exclude_unset=True).items():
setattr(user, field, value)
try:
await self.db.commit()
except IntegrityError as exc:
await self.db.rollback()
raise DuplicateUserError from exc
await self.db.refresh(user)
return user
async def authenticate(self, email: str, password: str) -> str | None:
user = await self.get_by_email(email)
if user is None or not pwd_context.verify(password, user.hashed_password):
return None
expire = datetime.now(timezone.utc) + timedelta(
minutes=settings.access_token_expire_minutes
)
return jwt.encode(
{"sub": str(user.id), "exp": expire},
settings.secret_key,
algorithm=settings.algorithm,
)
Note on Database Design: Application-level unique handling requires an underlying unique database index (e.g.,
unique=Trueon your SQLAlchemy mapping attributes). Without underlying constraints, application layer error-catching cannot safely prevent concurrent race conditions.
Testing with httpx and pytest
# tests/conftest.py
import pytest_asyncio
from httpx import ASGITransport, AsyncClient
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
from app.database import Base
from app.dependencies import get_db
from app.main import create_app
TEST_DATABASE_URL = "sqlite+aiosqlite:///:memory:"
engine = create_async_engine(TEST_DATABASE_URL)
TestingSessionLocal = async_sessionmaker(engine, expire_on_commit=False)
@pytest_asyncio.fixture(autouse=True)
async def setup_db():
async with engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
yield
async with engine.begin() as conn:
await conn.run_sync(Base.metadata.drop_all)
@pytest_asyncio.fixture
async def db_session():
async with TestingSessionLocal() as session:
yield session
await session.rollback()
@pytest_asyncio.fixture
async def client(db_session: AsyncSession):
app = create_app()
async def override_get_db():
yield db_session
app.dependency_overrides[get_db] = override_get_db
async with AsyncClient(
transport=ASGITransport(app=app), base_url="http://test"
) as ac:
yield ac
@pytest_asyncio.fixture
async def registered_user(client: AsyncClient) -> dict:
resp = await client.post("/users/", json={
"email": "test@example.com",
"username": "testuser",
"password": "securepass1",
"password_confirm": "securepass1",
})
assert resp.status_code == 201
return resp.json()
@pytest_asyncio.fixture
async def auth_token(client: AsyncClient, registered_user: dict) -> str:
resp = await client.post("/users/token", data={
"username": "test@example.com",
"password": "securepass1",
})
assert resp.status_code == 200
return resp.json()["access_token"]
@pytest_asyncio.fixture
async def auth_client(client: AsyncClient, auth_token: str) -> AsyncClient:
client.headers.update({"Authorization": f"Bearer {auth_token}"})
return client
Anti-Patterns
# Bad: business logic inside route handlers.
@router.post("/users/")
async def create_user(payload: UserCreate, db: DbDep):
hashed = bcrypt.hash(payload.password)
user = User(email=payload.email, hashed_password=hashed)
db.add(user)
await db.commit()
return user
# Good: thin route, transactional service handling.
@router.post("/users/", response_model=UserResponse, status_code=201)
async def create_user(payload: UserCreate, db: DbDep):
try:
return await UserService(db).create(payload)
except DuplicateUserError:
raise HTTPException(status_code=400, detail="Email already registered")
# Bad: sync DB calls in async routes block the event loop.
@router.get("/items/")
async def list_items(db: Session = Depends(get_db)):
return db.query(Item).all()
# Good: use async SQLAlchemy executions.
@router.get("/items/")
async def list_items(db: AsyncSession = Depends(get_db)):
result = await db.execute(select(Item))
return result.scalars().all()
Best Practices
- Always declare a typed
response_modelto prevent accidental PII/data leaks and output clean OpenAPI schemas. - Consolidate standard middleware dependency injections via type-aliasing:
DbDep = Annotated[AsyncSession, Depends(get_db)]. - Wrap database mutation boundaries gracefully within transactions inside your service layer, catching structural database errors directly.
- Parse JWT parameters defensively, expecting potential string/integer cast mismatches from modern payload variations.
- Enforce deterministic sorting (e.g.,
.order_by(Model.id)) on all offset/limit paginated endpoints to avoid data skips. - Isolate authorization checks from core authentication dependencies to provide precise REST status signals (
401vs403).
FastAPI Project Templates
Production-ready FastAPI project structures with async patterns, dependency injection, middleware, and best practices for building high-performance APIs.
When to Use This Skill
- Starting new FastAPI projects from scratch
- Implementing async REST APIs with Python
- Building high-performance web services and microservices
- Creating async applications with PostgreSQL, MongoDB
- Setting up API projects with proper structure and testing
Core Concepts
1. Project Structure
Recommended Layout:
app/
├── api/ # API routes
│ ├── v1/
│ │ ├── endpoints/
│ │ │ ├── users.py
│ │ │ ├── auth.py
│ │ │ └── items.py
│ │ └── router.py
│ └── dependencies.py # Shared dependencies
├── core/ # Core configuration
│ ├── config.py
│ ├── security.py
│ └── database.py
├── models/ # Database models
│ ├── user.py
│ └── item.py
├── schemas/ # Pydantic schemas
│ ├── user.py
│ └── item.py
├── services/ # Business logic
│ ├── user_service.py
│ └── auth_service.py
├── repositories/ # Data access
│ ├── user_repository.py
│ └── item_repository.py
└── main.py # Application entry
2. Dependency Injection
FastAPI's built-in DI system using Depends:
- Database session management
- Authentication/authorization
- Shared business logic
- Configuration injection
3. Async Patterns
Proper async/await usage:
- Async route handlers
- Async database operations
- Async background tasks
- Async middleware
4. Dependencies
The packages the examples import:
# requirements.txt
fastapi
uvicorn[standard]
pydantic-settings
sqlalchemy[asyncio]>=2.0 # DeclarativeBase; [asyncio] installs greenlet for the async engine
asyncpg # PostgreSQL driver: postgresql+asyncpg://...
PyJWT
pwdlib[argon2]
python-multipart # only for a login route that uses OAuth2PasswordRequestForm
# requirements-dev.txt
pytest
pytest-asyncio
httpx
aiosqlite # test database: sqlite+aiosqlite:///:memory:
Detailed worked examples and patterns
Detailed sections (starting with ## Implementation Patterns) live in references/details.md. Read that file when the navigation summary above is insufficient.
Testing
# tests/conftest.py
import pytest_asyncio
from httpx import ASGITransport, AsyncClient
from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession
from sqlalchemy.orm import sessionmaker
from app.main import app
from app.core.database import get_db, Base
TEST_DATABASE_URL = "sqlite+aiosqlite:///:memory:"
# Async fixtures need pytest_asyncio.fixture in pytest-asyncio's default strict mode
@pytest_asyncio.fixture
async def db_session():
engine = create_async_engine(TEST_DATABASE_URL, echo=True)
async with engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
AsyncSessionLocal = sessionmaker(
engine, class_=AsyncSession, expire_on_commit=False
)
async with AsyncSessionLocal() as session:
yield session
await engine.dispose()
@pytest_asyncio.fixture
async def client(db_session):
async def override_get_db():
yield db_session
app.dependency_overrides[get_db] = override_get_db
# httpx 0.28 removed AsyncClient(app=...); use an ASGI transport
transport = ASGITransport(app=app)
async with AsyncClient(transport=transport, base_url="http://test") as client:
yield client
app.dependency_overrides.clear()
# tests/test_users.py
import pytest
@pytest.mark.asyncio
async def test_create_user(client):
response = await client.post(
"/api/v1/users/",
json={
"email": "test@example.com",
"password": "testpass123",
"name": "Test User"
}
)
assert response.status_code == 201
data = response.json()
assert data["email"] == "test@example.com"
assert "id" in data
Generating Python Installer (Commercial-Grade)
You are a Python commercial deployment expert. Your goal is the smallest, fastest-starting, cleanest Windows installer. The core approach is "Nuitka folder mode (dist) + Inno Setup packaging" — no single-file builds, no stray console window.
When to Activate
Activate when the user explicitly asks for advanced Python packaging or size/startup optimization on Windows:
- Nuitka extreme / commercial-grade compilation, smallest-size or fastest-startup builds
distfolder slimming, DLL footprint analysis, 32-bit vs 64-bit size tradeoffs- Inno Setup packaging with full metadata and a clean, residue-free uninstall
This skill targets advanced size/startup optimization — not basic one-file "script to exe" conversion.
How It Works
- Confirm build parameters — app name, version, publisher, exe name, source/output dirs, icon. Never auto-fill; ask the user.
- Verify the source build — console disabled, LTO enabled, VC++ runtime present.
- Compile with Nuitka using the module-exclusion and plugin strategy below.
- Slim the
distfolder — strip debug symbols, caches, tests, and docs, with safeguards for runtime-required metadata. - Analyze DLLs to find and trim the largest dependencies.
- Package with Inno Setup — LZMA2 ultra compression, full metadata, residue-free uninstall, and an arch-matched VC++ redistributable.
Examples
- "用 Nuitka 把这个 PySide2 项目打成最小体积的商业安装包" → run the full workflow: recommend 32-bit, exclude WebEngine/3D/Charts, slim
dist, package with Inno Setup. - "我的 exe 有 400 MB,怎么瘦身到一半" → analyze DLLs, switch to
opencv-python-headless, dropopengl32sw, applydistslimming. - "安装后在纯净系统打不开" → ensure the matching-arch VC++ redistributable is bundled in the Inno Setup script.
核心理念
坚持 "Nuitka 文件夹模式(dist) + Inno Setup 封装" 方案。拒绝单文件版,拒绝黑窗。
实战参考案例(生产级 PySide2 桌面应用,323 MB,含 OpenCV / Playwright)
项目概况
- 总体积: 323 MB
- 打包工具: PyInstaller 4.7 (32位)
- 主要依赖: PySide2 (22.52 MB), OpenCV (62.38 MB), Playwright (76.74 MB)
- Python 版本: Python 3.8 (32位)
- DLL 数量: 71 个,总计 93.23 MB
关键优化策略
- PASS: 使用 32 位 Python → 体积减少 20-30%
- PASS: base_library.zip 压缩标准库 → 0.74 MB
- PASS: 精简模块排除 → 无 pytest/unittest/setuptools
- PASS: 精简 Qt 插件 → 只保留必要插件
体积分布
| 组件 | 体积 | 占比 | 优化建议 |
|---|---|---|---|
| playwright | 76.74 MB | 23.8% | 非必要可移除 |
| OpenCV | 62.38 MB | 19.3% | 用 opencv-python-headless |
| PySide2 | 22.52 MB | 7.0% | 排除 WebEngine/3D/Charts |
| 其他依赖 | 161.36 MB | 49.9% | - |
预期效果对比
| 项目类型 | Nuitka 原始 | 优化后 | 参考项目实测 |
|---|---|---|---|
| Tkinter + 标准库 | 150-250 MB | 80-120 MB | - |
| PyQt/PySide | 200-400 MB | 120-250 MB | 323 MB (含 OpenCV 等) |
| 含 numpy/pandas | 300-600 MB | 180-350 MB | - |
核心工作流 (Workflow) - WARNING: 严格执行
当用户请求打包时,按照以下步骤操作:
步骤 1:强制参数确认(FAIL: 禁止使用默认值)
WARNING: 重要规则:以下所有参数必须逐一向用户确认,禁止自动填充或使用默认值!
必须向用户询问并确认以下信息(等待用户明确回复后才能继续):
| 参数 | 说明 | 示例 |
|---|---|---|
| 软件名称 (App Name) | 软件显示名称 | 红墨批注 |
| 版本号 (Version) | 语义化版本号 | 1.0.0 |
| 发布者/公司名 (Publisher) | 控制面板显示的发布者 | YourCompany |
| 主程序 (Exe Name) | 主可执行文件名 | RedInk.exe |
| 源路径 (Source Dir) | Nuitka dist 文件夹绝对路径 | D:\project\dist |
| 输出路径 (Output Dir) | 安装包生成位置 | D:\project\output |
| 图标路径 (Icon Path) | .ico 文件绝对路径(可选但推荐) | D:\project\icon.ico |
| 官网地址 (URL) | 可选,用于控制面板链接 | https://example.com |
询问模板:
"请提供以下打包参数,我需要您逐一确认:
- 软件名称:
- 版本号:
- 发布者/公司名:
- 主程序文件名(如 xxx.exe):
- 源路径(Nuitka dist 文件夹):
- 输出路径(安装包保存位置):
- 图标路径(.ico 文件,可留空):
- 官网地址(可留空):
请逐一填写,或回复"跳过"表示使用空值。"
步骤 2:源文件质量与编译检查(关键) 在生成代码之前,必须向用户发出以下关键确认(因为 Inno Setup 只是打包工具,无法改变程序本身的运行属性):
"WARNING: 编译参数检查:
- 去黑窗:请确认您的 dist 文件夹是使用
nuitka --windows-console-mode=disable编译的。(否则安装后依然会有黑框)- 高性能:请确认是否使用了
--lto=yes。(否则启动速度可能不理想)- 运行库:请确保 dist 文件夹内已包含必要的 VC++ 运行库,防止在纯净系统上无法运行。
确认源文件已准备好请回复"确认",否则请先重新编译。"
步骤 3:生成代码 用户确认后,输出包含 完整元数据 和 卸载图标修复 的代码。
Nuitka 极限优化编译(基于 参考项目经验)
一、32 位 vs 64 位选择策略
参考项目使用 32 位 Python 的原因:
| 组件 | 64位体积 | 32位体积 | 节省 |
|---|---|---|---|
| python3x.dll | ~4.5 MB | ~3.8 MB | 15% |
| Qt5Core.dll | ~8 MB | ~5 MB | 37% |
| numpy | ~30 MB | ~20 MB | 33% |
| 总体 | 基准 | -20~30% | - |
推荐使用 32 位条件:
- PASS: 程序内存占用 < 2GB
- PASS: 不处理超大文件(< 2GB)
- PASS: 目标用户是普通办公电脑
32 位编译方法:
# 1. 安装 32 位 Python(和 64 位可以共存)
# 下载地址:https://www.python.org/downloads/windows/
# 2. 用 32 位 Python 安装依赖
py -3.12-32 -m pip install -r requirements.txt
# 3. 用 32 位 Python 编译
py -3.12-32 -m nuitka --standalone ...你的参数
二、模块排除清单(参考项目验证过的)
安全排除列表(运行时不需要):
unittest,test,pytest,_pytest,doctest,pdb,pdbpp,
setuptools,pip,distutils,pkg_resources,
email.mime,http.server,xmlrpc,pydoc
预期效果:节省 30-50 MB
三、GUI 框架专用优化
Tkinter 极限优化(推荐,最轻量)
nuitka --standalone --windows-console-mode=disable ^
--lto=yes ^
--jobs=8 ^
--enable-plugin=tk-inter ^
--enable-plugin=anti-bloat ^
--noinclude-pytest-mode=nofollow ^
--noinclude-setuptools-mode=nofollow ^
--nofollow-import-to=unittest,test,pytest,_pytest,doctest,pdb,pdbpp ^
--nofollow-import-to=setuptools,pip,distutils,pkg_resources ^
--nofollow-import-to=email.mime,http.server,xmlrpc,pydoc ^
--python-flag=no_docstrings ^
--output-dir=dist ^
--windows-icon-from-ico=icon.ico ^
--remove-output ^
main.py
预期体积:80-120 MB(优化后)
PyQt5 / PySide2 优化
nuitka --standalone --windows-console-mode=disable ^
--lto=yes ^
--jobs=8 ^
--enable-plugin=pyqt5 ^
--enable-plugin=anti-bloat ^
--noinclude-pytest-mode=nofollow ^
--noinclude-setuptools-mode=nofollow ^
--nofollow-import-to=unittest,test,pytest,_pytest,doctest,pdb ^
--nofollow-import-to=setuptools,pip,distutils,pkg_resources ^
--nofollow-import-to=PyQt5.QtWebEngine,PyQt5.QtWebEngineWidgets ^
--nofollow-import-to=PyQt5.Qt3D,PyQt5.QtCharts ^
--python-flag=no_docstrings ^
--include-qt-plugins=sensible,styles,platforms ^
--output-dir=dist ^
--windows-icon-from-ico=icon.ico ^
--remove-output ^
main.py
预期体积:120-250 MB(优化后)
四、一键编译脚本模板
保存为 build_optimized.bat(项目根目录):
@echo off
chcp 65001 >nul
setlocal enabledelayedexpansion
echo ========================================
echo Nuitka 极限优化编译(参考 参考项目)
echo ========================================
REM === 配置区域(请修改为你的实际值) ===
set APP_NAME=你的软件名
set MAIN_FILE=main.py
set ICON_FILE=icon.ico
REM === 自动检测 CPU 核心数 ===
REM 用 Windows 自带环境变量(wmic 在 Win11 22H2+ 已移除,探不到会让 --jobs=0 单线程编译)
set CPU_CORES=%NUMBER_OF_PROCESSORS%
if not defined CPU_CORES set CPU_CORES=4
set /a BUILD_JOBS=%CPU_CORES%
REM === 参考项目的模块排除清单 ===
set EXCLUDE_MODULES=unittest,test,pytest,_pytest,doctest,pdb,pdbpp
set EXCLUDE_MODULES=%EXCLUDE_MODULES%,setuptools,pip,distutils,pkg_resources
set EXCLUDE_MODULES=%EXCLUDE_MODULES%,email.mime,http.server,xmlrpc,pydoc
echo.
echo [1/4] 清理旧编译...
if exist dist rd /s /q dist
if exist build rd /s /q build
echo.
echo [2/4] Nuitka 编译中(应用 参考项目优化策略)...
echo - CPU 核心: %CPU_CORES% (使用 %BUILD_JOBS% 线程)
echo - 模块排除: %EXCLUDE_MODULES%
echo.
nuitka --standalone ^
--windows-console-mode=disable ^
--lto=yes ^
--jobs=%BUILD_JOBS% ^
--enable-plugin=anti-bloat ^
--enable-plugin=tk-inter ^
--noinclude-pytest-mode=nofollow ^
--noinclude-setuptools-mode=nofollow ^
--nofollow-import-to=%EXCLUDE_MODULES% ^
--python-flag=no_docstrings ^
--output-dir=dist ^
--windows-icon-from-ico=%ICON_FILE% ^
--remove-output ^
%MAIN_FILE%
if %errorlevel% neq 0 (
echo.
echo [错误] 编译失败!
pause
exit /b 1
)
echo.
echo [3/4] 统计编译结果...
for /f %%a in ('powershell -NoProfile -Command "(Get-ChildItem -LiteralPath 'dist\%APP_NAME%.dist' -Recurse -File | Measure-Object -Property Length -Sum).Sum"') do set TOTAL_SIZE=%%a
set TOTAL_SIZE=%TOTAL_SIZE:,=%
set /a SIZE_MB=%TOTAL_SIZE% / 1048576
echo - 编译后体积: %SIZE_MB% MB
echo.
echo [4/4] 执行瘦身清理(参考 参考项目策略)...
powershell -ExecutionPolicy Bypass -File slim_dist.ps1 -DistPath "dist\%APP_NAME%.dist"
echo.
echo ========================================
echo 编译完成!
echo ========================================
pause
五、dist 瘦身脚本(参考项目级别清理)
保存为 slim_dist.ps1(和 build_optimized.bat 同目录):
param(
[string]$DistPath
)
$ErrorActionPreference = "Continue" # 不静默吞错:删除失败会显示出来,避免假成功
Write-Host "`n========================================" -ForegroundColor Cyan
Write-Host "dist 瘦身清理(参考 参考项目策略)" -ForegroundColor Cyan
Write-Host "========================================" -ForegroundColor Cyan
if (-not (Test-Path $DistPath)) {
Write-Host "[错误] 找不到目录: $DistPath" -ForegroundColor Red
exit 1
}
# 统计初始体积
$InitialSize = (Get-ChildItem -Path $DistPath -Recurse -File | Measure-Object -Property Length -Sum).Sum / 1MB
Write-Host "`n初始体积: $([math]::Round($InitialSize, 2)) MB" -ForegroundColor Yellow
# 参考项目特征:没有 .pdb, .pyi, __pycache__, test 等
Write-Host "`n[应用 参考项目的清理策略...]" -ForegroundColor Green
# 1. 删除调试符号
Write-Host "`n[1/7] 删除 .pdb 调试符号..." -ForegroundColor Green
$pdbFiles = Get-ChildItem -Path $DistPath -Recurse -Include *.pdb -File
$pdbSize = ($pdbFiles | Measure-Object -Property Length -Sum).Sum / 1MB
if ($pdbFiles.Count -gt 0) {
$pdbFiles | Remove-Item -Force
Write-Host " 删除 $($pdbFiles.Count) 个文件,节省 $([math]::Round($pdbSize, 2)) MB"
} else {
Write-Host " 未发现 .pdb 文件(已优化)" -ForegroundColor Gray
}
# 2. 删除类型提示
Write-Host "`n[2/7] 删除 .pyi 类型提示..." -ForegroundColor Green
$pyiFiles = Get-ChildItem -Path $DistPath -Recurse -Include *.pyi -File
$pyiSize = ($pyiFiles | Measure-Object -Property Length -Sum).Sum / 1MB
if ($pyiFiles.Count -gt 0) {
$pyiFiles | Remove-Item -Force
Write-Host " 删除 $($pyiFiles.Count) 个文件,节省 $([math]::Round($pyiSize, 2)) MB"
} else {
Write-Host " 未发现 .pyi 文件(已优化)" -ForegroundColor Gray
}
# 3. 删除 __pycache__
Write-Host "`n[3/7] 删除 __pycache__ 缓存..." -ForegroundColor Green
$pycacheDirs = Get-ChildItem -Path $DistPath -Recurse -Directory -Filter "__pycache__"
$pycacheSize = 0
foreach ($dir in $pycacheDirs) {
$size = (Get-ChildItem -Path $dir.FullName -Recurse -File | Measure-Object -Property Length -Sum).Sum
$pycacheSize += $size
Remove-Item -Path $dir.FullName -Recurse -Force
}
if ($pycacheDirs.Count -gt 0) {
Write-Host " 删除 $($pycacheDirs.Count) 个目录,节省 $([math]::Round($pycacheSize / 1MB, 2)) MB"
} else {
Write-Host " 未发现 __pycache__(已优化)" -ForegroundColor Gray
}
# 4. 删除测试目录
Write-Host "`n[4/7] 删除 test/tests 测试目录..." -ForegroundColor Green
$testDirs = Get-ChildItem -Path $DistPath -Recurse -Directory | Where-Object { $_.Name -match '^tests?$' }
$testSize = 0
foreach ($dir in $testDirs) {
$size = (Get-ChildItem -Path $dir.FullName -Recurse -File | Measure-Object -Property Length -Sum).Sum
$testSize += $size
Remove-Item -Path $dir.FullName -Recurse -Force
}
if ($testDirs.Count -gt 0) {
Write-Host " 删除 $($testDirs.Count) 个目录,节省 $([math]::Round($testSize / 1MB, 2)) MB"
} else {
Write-Host " 未发现测试目录(已优化)" -ForegroundColor Gray
}
# 5. 删除文档和示例
Write-Host "`n[5/7] 删除 docs/examples 文档目录..." -ForegroundColor Green
$docDirs = Get-ChildItem -Path $DistPath -Recurse -Directory | Where-Object { $_.Name -match '^(docs|examples|samples|demo)$' }
$docSize = 0
foreach ($dir in $docDirs) {
$size = (Get-ChildItem -Path $dir.FullName -Recurse -File | Measure-Object -Property Length -Sum).Sum
$docSize += $size
Remove-Item -Path $dir.FullName -Recurse -Force
}
if ($docDirs.Count -gt 0) {
Write-Host " 删除 $($docDirs.Count) 个目录,节省 $([math]::Round($docSize / 1MB, 2)) MB"
} else {
Write-Host " 未发现文档目录(已优化)" -ForegroundColor Gray
}
# 6. 删除 .pyc 文件
Write-Host "`n[6/7] 删除 .pyc 字节码..." -ForegroundColor Green
$pycFiles = Get-ChildItem -Path $DistPath -Recurse -Include *.pyc -File
$pycSize = ($pycFiles | Measure-Object -Property Length -Sum).Sum / 1MB
if ($pycFiles.Count -gt 0) {
$pycFiles | Remove-Item -Force
Write-Host " 删除 $($pycFiles.Count) 个文件,节省 $([math]::Round($pycSize, 2)) MB"
} else {
Write-Host " 未发现 .pyc 文件(已优化)" -ForegroundColor Gray
}
# 7. 精简 .dist-info 元数据
Write-Host "`n[7/7] 精简 .dist-info 元数据..." -ForegroundColor Green
$distInfoDirs = Get-ChildItem -Path $DistPath -Recurse -Directory -Filter "*.dist-info"
$removedCount = 0
$removedSize = 0
foreach ($infoDir in $distInfoDirs) {
# 仅删安装期记账文件;保留 METADATA 与 entry_points.txt(运行期被 importlib.metadata 读取,删除会破坏插件发现)
$filesToRemove = @("RECORD", "INSTALLER", "direct_url.json")
foreach ($fileName in $filesToRemove) {
$file = Join-Path $infoDir.FullName $fileName
if (Test-Path $file) {
$size = (Get-Item $file).Length
$removedSize += $size
Remove-Item $file -Force
$removedCount++
}
}
}
if ($removedCount -gt 0) {
Write-Host " 删除 $removedCount 个元数据文件,节省 $([math]::Round($removedSize / 1MB, 2)) MB"
} else {
Write-Host " 未发现可清理的元数据(已优化)" -ForegroundColor Gray
}
# 统计最终体积
$FinalSize = (Get-ChildItem -Path $DistPath -Recurse -File | Measure-Object -Property Length -Sum).Sum / 1MB
$SavedSize = $InitialSize - $FinalSize
$SavedPercent = if ($InitialSize -gt 0) { ($SavedSize / $InitialSize) * 100 } else { 0 }
Write-Host "`n========================================" -ForegroundColor Cyan
Write-Host "清理完成!" -ForegroundColor Green
Write-Host "========================================" -ForegroundColor Cyan
Write-Host "初始体积: $([math]::Round($InitialSize, 2)) MB" -ForegroundColor Yellow
Write-Host "最终体积: $([math]::Round($FinalSize, 2)) MB" -ForegroundColor Green
Write-Host "节省空间: $([math]::Round($SavedSize, 2)) MB ($([math]::Round($SavedPercent, 1))%)" -ForegroundColor Cyan
Write-Host "========================================`n" -ForegroundColor Cyan
# 对比 参考项目
Write-Host "[对比参考]" -ForegroundColor Yellow
Write-Host "参考项目总体积: 323 MB (包含 PyQt, OpenCV, Playwright 等重量级库)" -ForegroundColor Gray
Write-Host "如果你的项目是纯 Tkinter + 标准库,目标应该在 80-150 MB" -ForegroundColor Gray
预期效果:节省 15-30% 体积
六、DLL 依赖分析工具
保存为 analyze_dlls.py(用于找出体积大户):
"""
DLL 依赖分析工具
参考 参考项目的 DLL 管理策略,帮助识别体积大户和优化建议
"""
import sys
from pathlib import Path
def analyze_dlls(dist_path: str):
"""分析 dist 目录中的 DLL 依赖"""
dist_dir = Path(dist_path)
if not dist_dir.exists():
print(f"[错误] 目录不存在: {dist_path}")
return
print("=" * 70)
print("DLL 依赖分析(参考 参考项目策略)")
print("=" * 70)
# 收集所有 DLL
dll_files = list(dist_dir.rglob("*.dll"))
if not dll_files:
print("\n未发现 DLL 文件")
return
# 按大小排序
dll_data = [(dll, dll.stat().st_size) for dll in dll_files]
dll_data.sort(key=lambda x: x[1], reverse=True)
total_size = sum(size for _, size in dll_data)
print(f"\n总 DLL 数量: {len(dll_files)}")
print(f"总 DLL 体积: {total_size / 1024 / 1024:.2f} MB\n")
# 参考项目对比
print("[对比参考] 参考项目的 DLL 情况:")
print(" - 总数量: 71 个")
print(" - 总体积: 93.23 MB")
print(" - 最大的: libopenblas (26.85 MB), opengl32sw (15.25 MB)\n")
# 分析大于 3MB 的 DLL
large_dlls = [(dll, size) for dll, size in dll_data if size > 3 * 1024 * 1024]
if large_dlls:
print("=" * 70)
print("WARNING: 大于 3MB 的 DLL(需重点关注)")
print("=" * 70)
for dll, size in large_dlls:
size_mb = size / 1024 / 1024
relative_path = dll.relative_to(dist_dir)
name_lower = dll.name.lower()
print(f"\n{size_mb:8.2f} MB {dll.name}")
print(f" 位置: {relative_path.parent}")
# 优化建议
suggestions = get_optimization_suggestion(name_lower)
if suggestions:
for suggestion in suggestions:
print(f" {suggestion}")
# 检查冗余 DLL
print("\n" + "=" * 70)
print(" 冗余检查")
print("=" * 70)
# 检查调试版本
debug_dlls = [dll for dll, _ in dll_data if dll.stem.endswith('d')]
if debug_dlls:
print(f"\nWARNING: 发现 {len(debug_dlls)} 个调试版本 DLL(可以删除):")
for dll in debug_dlls:
print(f" - {dll.name}")
else:
print("\nPASS: 未发现调试版本 DLL(已优化)")
# VC++ Runtime
vc_runtimes = [dll for dll, _ in dll_data if 'vcruntime' in dll.name.lower() or 'msvcp' in dll.name.lower()]
if vc_runtimes:
print(f"\n[VC++ Runtime 库] 发现 {len(vc_runtimes)} 个:")
for dll in vc_runtimes:
size_mb = dll.stat().st_size / 1024 / 1024
print(f" - {dll.name} ({size_mb:.2f} MB)")
print(" 这些是必需的,参考项目也包含了这些文件")
# 全部 DLL 列表
print("\n" + "=" * 70)
print(" 完整 DLL 列表(按体积排序,前 20)")
print("=" * 70)
print(f"\n{'体积 (MB)':>10} {'文件名':<30} 位置")
print("-" * 70)
for dll, size in dll_data[:20]:
size_mb = size / 1024 / 1024
relative_path = dll.relative_to(dist_dir)
location = str(relative_path.parent) if relative_path.parent != Path('.') else "根目录"
print(f"{size_mb:10.2f} {dll.name:<30} {location}")
if len(dll_data) > 20:
remaining_size = sum(size for _, size in dll_data[20:]) / 1024 / 1024
print(f"... 还有 {len(dll_data) - 20} 个 DLL,共 {remaining_size:.2f} MB")
def get_optimization_suggestion(dll_name: str) -> list:
"""根据 DLL 名称给出优化建议"""
suggestions = []
if "openblas" in dll_name or "mkl" in dll_name:
suggestions.append("数学运算库,参考项目的 libopenblas 有 26.85 MB")
suggestions.append("如不需要高性能计算可考虑轻量版")
elif "opencv" in dll_name or "ffmpeg" in dll_name:
suggestions.append("OpenCV 相关,参考项目的 opencv_videoio_ffmpeg 有 18.48 MB")
suggestions.append("考虑用 opencv-python-headless")
elif "qt5" in dll_name or "qt6" in dll_name or "pyside" in dll_name:
suggestions.append("Qt 库,参考项目的 Qt5Core 有 5.13 MB")
suggestions.append("可排除不需要的模块(WebEngine, 3D, Charts)")
elif "opengl" in dll_name and "sw" in dll_name:
suggestions.append("OpenGL 软件渲染器,参考项目保留了 15.25 MB")
suggestions.append("通常可以删除(使用硬件渲染)")
elif "d3dcompiler" in dll_name:
suggestions.append("DirectX 编译器,参考项目有 3.53 MB")
elif "mfc140" in dll_name:
suggestions.append("MFC 库,参考项目有 4.89 MB")
return suggestions
if __name__ == "__main__":
if len(sys.argv) < 2:
print("用法: python analyze_dlls.py <dist目录路径>")
print("示例: python analyze_dlls.py dist/main.dist")
sys.exit(1)
analyze_dlls(sys.argv[1])
使用方法:
python analyze_dlls.py dist/你的软件名.dist
完整优化工作流
步骤 1:修改编译脚本配置
编辑 build_optimized.bat,修改这 3 行:
set APP_NAME=你的软件名 REM 改成实际名称
set MAIN_FILE=main.py REM 你的主程序文件
set ICON_FILE=icon.ico REM 你的图标文件
步骤 2:一键编译+瘦身
# 在项目根目录执行
build_optimized.bat
步骤 3:分析 DLL 依赖
python analyze_dlls.py dist/你的软件名.dist
步骤 4:根据分析结果优化
如果用了 OpenCV → 改用无头版
pip uninstall opencv-python
pip install opencv-python-headless
如果用了 Qt → 排除不需要的模块
# 在编译命令中添加
--nofollow-import-to=PyQt5.QtWebEngine,PyQt5.Qt3D,PyQt5.QtCharts
删除软件渲染器(如果不需要)
# 在 dist 目录执行
Remove-Item "opengl32sw.dll" -Force
VC++ 运行库处理方案
方案一:静态链接(推荐)
nuitka --static-libpython=yes ...
方案二:捆绑运行库安装(商业发布推荐)
在 Inno Setup 脚本中添加:
; WARNING: VC++ 运行库架构必须与 Python/Nuitka 构建架构一致。
; 本 skill 推荐 32 位 Python,故默认捆绑 vc_redist.x86.exe;
; 若用 64 位 Python 编译,请把下面两处改为 vc_redist.x64.exe。
[Files]
Source: "{#MySourceDir}\..\vc_redist.x86.exe"; DestDir: "{tmp}"; Flags: deleteafterinstall
[Run]
Filename: "{tmp}\vc_redist.x86.exe"; Parameters: "/quiet /norestart"; StatusMsg: "正在安装运行库..."; Flags: waituntilterminated
Inno Setup 脚本模板(商业终极版)
; =====================================================================
; WARNING: 商业级 Python 安装脚本 (Inno Setup 6.x)
; 特性:LZMA2 极限压缩 | 全中文 | 完整元数据 | 无残留卸载
; 参考:参考项目(323 MB, LZMA2 压缩)
; =====================================================================
; --- 1. 参数定义 ---
#define MyAppName "{{APP_NAME}}"
#define MyAppVersion "{{APP_VERSION}}"
#define MyAppPublisher "{{PUBLISHER}}"
#define MyAppURL "{{APP_URL}}"
#define MyAppExeName "{{EXE_NAME}}"
#define MySourceDir "{{SOURCE_DIR}}"
#define MyOutputDir "{{OUTPUT_DIR}}"
;#define MyIconPath "{{ICON_PATH}}"
[Setup]
; --- 身份识别 ---
AppId={{GENERATE_RANDOM_GUID}}
AppName={#MyAppName}
AppVersion={#MyAppVersion}
AppPublisher={#MyAppPublisher}
AppPublisherURL={#MyAppURL}
AppSupportURL={#MyAppURL}
AppUpdatesURL={#MyAppURL}
; --- 安装路径与权限 ---
DefaultDirName={autopf}\{#MyAppName}
DefaultGroupName={#MyAppName}
DisableDirPage=no
DisableProgramGroupPage=no
PrivilegesRequired=admin
; --- 输出设置 ---
OutputDir={#MyOutputDir}
OutputBaseFilename=Setup_{#MyAppName}_v{#MyAppVersion}
; --- 视觉体验 ---
WizardStyle=modern
#ifdef MyIconPath
SetupIconFile={#MyIconPath}
UninstallDisplayIcon={app}\{#MyAppExeName}
#endif
; --- 核心压缩 (参考 参考项目) ---
Compression=lzma2/ultra64
SolidCompression=yes
LZMAUseSeparateProcess=yes
; --- 架构 ---
; 注意:仅 64 位 Python 构建才设此项。本 skill 推荐 32 位 Python——
; 32 位构建请保持注释,使应用按 32 位安装并与上面捆绑的 vc_redist.x86.exe 匹配。
; 仅当用 64 位 Python 编译时才取消注释。
;ArchitecturesInstallIn64BitMode=x64compatible
[Languages]
Name: "chinesesimplified"; MessagesFile: "compiler:Languages\ChineseSimplified.isl"
[Files]
Source: "{#MySourceDir}\*"; DestDir: "{app}"; Flags: ignoreversion recursesubdirs createallsubdirs
[UninstallDelete]
Type: filesandordirs; Name: "{app}\*"
[Icons]
Name: "{autodesktop}\{#MyAppName}"; Filename: "{app}\{#MyAppExeName}"; Tasks: desktopicon
Name: "{group}\{#MyAppName}"; Filename: "{app}\{#MyAppExeName}"
Name: "{group}\卸载 {#MyAppName}"; Filename: "{uninstallexe}"
[Tasks]
Name: "desktopicon"; Description: "{cm:CreateDesktopIcon}"; GroupDescription: "{cm:AdditionalIcons}"; Flags: unchecked
[Run]
Filename: "{app}\{#MyAppExeName}"; Description: "{cm:LaunchProgram,{#MyAppName}}"; Flags: nowait postinstall skipifsilent
占位符说明
| 占位符 | 说明 | 示例值 |
|---|---|---|
{{APP_NAME}} | 软件显示名称 | 红墨批注 |
{{APP_VERSION}} | 版本号 | 1.0.0 |
{{PUBLISHER}} | 发布者/公司名 | MyCompany |
{{APP_URL}} | 官网地址 | https://example.com |
{{EXE_NAME}} | 主程序文件名 | RedInk.exe |
{{SOURCE_DIR}} | Nuitka dist 文件夹路径 | D:\project\dist\RedInk.dist |
{{OUTPUT_DIR}} | 安装包输出路径 | D:\project\output |
{{ICON_PATH}} | 图标文件路径 | D:\project\icon.ico |
{{GENERATE_RANDOM_GUID}} | 需生成唯一GUID | 使用 Inno Setup "Tools > Generate GUID" |
常见问题 (FAQ)
Q1: 安装后双击程序无反应?
- 打开 CMD,手动运行 exe 查看错误信息
- 检查是否缺少 VC++ 运行库
- 检查 Nuitka 编译是否成功
Q2: 安装包体积过大?
优化方法:
- 使用 32 位 Python 编译(节省 20-30%)
- 应用 参考项目的模块排除清单
- 启用 Anti-Bloat 插件
- 执行 dist 文件夹瘦身脚本
- 分析 DLL,移除不必要的大文件
Q3: 杀毒软件误报?
解决方案:
- 提交到主流杀毒厂商进行白名单申请
- 购买代码签名证书(推荐:Sectigo, DigiCert)
- 避免使用 UPX 压缩
Q4: 安装时提示 Windows 已保护你的电脑?
- 购买 EV 代码签名证书(可立即获得信任)
- 普通代码签名证书需要积累安装量后逐渐获得信任
实战问题处理记录(更新 2026-02-07)
- 安装后提示缺少 python3xx.dll:必须使用 Nuitka
--standalone;确认 dist 内存在该 dll;不要打单文件版。 - 安装后点击无反应:GUI 启动期可能被重依赖阻塞;将重依赖延迟到"开始导出"再 import;添加日志排查。
- Nuitka + MinGW 在非 ASCII 路径报错:把源码复制到 ASCII 目录再编译;设置
PYTHONIOENCODING=utf-8。 - Inno Setup 警告
x64已弃用(仅 64 位构建需要 64 位安装模式时):改为ArchitecturesInstallIn64BitMode=x64compatible;32 位构建无需此项。 --disable-console已废弃:改用--windows-console-mode=disable。- dist 出现
_nuitka_temp.exe:在 [Files] 中排除它。
优化效果预期
| 优化组合 | 体积减少 | 启动提升 | 风险等级 |
|---|---|---|---|
| 基础编译 | 基准 | 基准 | 无 |
+ --lto=yes | 5-10% | 10-20% | PASS: 无 |
| + anti-bloat | 15-25% | - | PASS: 无 |
| + 模块排除 | 20-35% | 5% | PASS: 无 |
| + dist 瘦身 | 25-40% | - | PASS: 无 |
| + 32 位编译 | 40-60% | - | PASS: 无 |
| 全部组合 | 45-65% | 15-25% | PASS: 无风险 |
WARNING: 不建议使用 UPX 压缩,虽然能进一步减小体积,但极易触发杀毒软件误报。
基于 参考项目实战经验优化,助你打造商业级安装包!
Modern Python
Guide for modern Python tooling and best practices, based on trailofbits/cookiecutter-python.
When to Use This Skill
- Creating a new Python project or package
- Setting up
pyproject.tomlconfiguration - Configuring development tools (linting, formatting, testing)
- Writing Python scripts with external dependencies
- Migrating from legacy tools (when user requests it)
When NOT to Use This Skill
- User wants to keep legacy tooling: Respect existing workflows if explicitly requested
- Python < 3.11 required: These tools target modern Python
- Non-Python projects: Mixed codebases where Python isn't primary
Anti-Patterns to Avoid
| Avoid | Use Instead |
|---|---|
[tool.ty] python-version | [tool.ty.environment] python-version |
uv pip install | uv add and uv sync |
| Editing pyproject.toml manually to add deps | uv add <pkg> / uv remove <pkg> |
hatchling build backend | uv_build (simpler, sufficient for most cases) |
| Poetry | uv (faster, simpler, better ecosystem integration) |
| requirements.txt | PEP 723 for scripts, pyproject.toml for projects |
| mypy / pyright | ty (faster, from Astral team) |
[project.optional-dependencies] for dev tools | [dependency-groups] (PEP 735) |
Manual virtualenv activation (source .venv/bin/activate) | uv run <cmd> |
| pre-commit | prek (faster, no Python runtime needed) |
Key principles:
- Always use
uv addanduv removeto manage dependencies - Never manually activate or manage virtual environments—use
uv runfor all commands - Use
[dependency-groups]for dev/test/docs dependencies, not[project.optional-dependencies]
Decision Tree
What are you doing?
│
├─ Single-file script with dependencies?
│ └─ Use PEP 723 inline metadata (./references/pep723-scripts.md)
│
├─ New multi-file project (not distributed)?
│ └─ Minimal uv setup (see Quick Start below)
│
├─ New reusable package/library?
│ └─ Full project setup (see Full Setup below)
│
└─ Migrating existing project?
└─ See Migration Guide below
Tool Overview
| Tool | Purpose | Replaces |
|---|---|---|
| uv | Package/dependency management | pip, virtualenv, pip-tools, pipx, pyenv |
| ruff | Linting AND formatting | flake8, black, isort, pyupgrade, pydocstyle |
| ty | Type checking | mypy, pyright (faster alternative) |
| pytest | Testing with coverage | unittest |
| prek | Pre-commit hooks (setup) | pre-commit (faster, Rust-native) |
Security Tools
| Tool | Purpose | When It Runs |
|---|---|---|
| shellcheck | Shell script linting | pre-commit |
| detect-secrets | Secret detection | pre-commit |
| actionlint | Workflow syntax validation | pre-commit, CI |
| zizmor | Workflow security audit | pre-commit, CI |
| pip-audit | Dependency vulnerability scanning | CI, manual |
| Dependabot | Automated dependency updates | scheduled |
See security-setup.md for configuration and usage.
Quick Start: Minimal Project
For simple multi-file projects not intended for distribution:
# Create project with uv
uv init myproject
cd myproject
# Add dependencies
uv add requests rich
# Add dev dependencies
uv add --group dev pytest ruff ty
# Run code
uv run python src/myproject/main.py
# Run tools
uv run pytest
uv run ruff check .
Full Project Setup
If starting from scratch, ask the user if they prefer to use the Trail of Bits cookiecutter template to bootstrap a complete project with already preconfigured tooling.
uvx cookiecutter gh:trailofbits/cookiecutter-python
1. Create Project Structure
uv init --package myproject
cd myproject
This creates:
myproject/
├── pyproject.toml
├── README.md
├── src/
│ └── myproject/
│ └── __init__.py
└── .python-version
2. Configure pyproject.toml
See pyproject.md for complete configuration reference.
Key sections:
[project]
name = "myproject"
version = "0.1.0"
requires-python = ">=3.11"
dependencies = []
[dependency-groups]
dev = [{include-group = "lint"}, {include-group = "test"}, {include-group = "audit"}]
lint = ["ruff", "ty"]
test = ["pytest", "pytest-cov"]
audit = ["pip-audit"]
[tool.ruff]
line-length = 100
target-version = "py311"
[tool.ruff.lint]
select = ["ALL"]
ignore = ["D", "COM812", "ISC001"]
[tool.pytest]
addopts = ["--cov=myproject", "--cov-fail-under=80"]
[tool.ty.terminal]
error-on-warning = true
[tool.ty.environment]
python-version = "3.11"
[tool.ty.rules]
# Strict from day 1 for new projects
possibly-unresolved-reference = "error"
unused-ignore-comment = "warn"
3. Install Dependencies
# Install all dependency groups
uv sync --all-groups
# Or install specific groups
uv sync --group dev
4. Add Makefile
.PHONY: dev lint format test build
dev:
uv sync --all-groups
lint:
uv run ruff format --check && uv run ruff check && uv run ty check src/
format:
uv run ruff format .
test:
uv run pytest
build:
uv build
Migration Guide
When a user requests migration from legacy tooling:
From requirements.txt + pip
First, determine the nature of the code:
For standalone scripts: Convert to PEP 723 inline metadata (see pep723-scripts.md)
For projects:
# Initialize uv in existing project
uv init --bare
# Add dependencies using uv (not by editing pyproject.toml)
uv add requests rich # add each package
# Or import from requirements.txt (review each package before adding)
# Note: Complex version specifiers may need manual handling
grep -v '^#' requirements.txt | grep -v '^-' | grep -v '^\s*$' | while read -r pkg; do
uv add "$pkg" || echo "Failed to add: $pkg"
done
uv sync
Then:
- Delete
requirements.txt,requirements-dev.txt - Delete virtual environment (
venv/,.venv/) - Add
uv.lockto version control
From setup.py / setup.cfg
- Run
uv init --bareto create pyproject.toml - Use
uv addto add each dependency frominstall_requires - Use
uv add --group devfor dev dependencies - Copy non-dependency metadata (name, version, description, etc.) to
[project] - Delete
setup.py,setup.cfg,MANIFEST.in
From flake8 + black + isort
- Remove flake8, black, isort via
uv remove - Delete
.flake8,pyproject.toml [tool.black],[tool.isort]configs - Add ruff:
uv add --group dev ruff - Add ruff configuration (see ruff-config.md)
- Run
uv run ruff check --fix .to apply fixes - Run
uv run ruff format .to format
From mypy / pyright
- Remove mypy/pyright via
uv remove - Delete
mypy.ini,pyrightconfig.json, or[tool.mypy]/[tool.pyright]sections - Add ty:
uv add --group dev ty - Run
uv run ty check src/
Quick Reference: uv Commands
| Command | Description |
|---|---|
uv init | Create new project |
uv init --package | Create distributable package |
uv add <pkg> | Add dependency |
uv add --group dev <pkg> | Add to dependency group |
uv remove <pkg> | Remove dependency |
uv sync | Install dependencies |
uv sync --all-groups | Install all dependency groups |
uv run <cmd> | Run command in venv |
uv run --with <pkg> <cmd> | Run with temporary dependency |
uv build | Build package |
uv publish | Publish to PyPI |
Ad-hoc Dependencies with --with
Use uv run --with for one-off commands that need packages not in your project:
# Run Python with a temporary package
uv run --with requests python -c "import requests; print(requests.get('https://httpbin.org/ip').json())"
# Run a module with temporary deps
uv run --with rich python -m rich.progress
# Multiple packages
uv run --with requests --with rich python script.py
# Combine with project deps (adds to existing venv)
uv run --with httpx pytest # project deps + httpx
When to use --with vs uv add:
uv add: Package is a project dependency (goes in pyproject.toml/uv.lock)--with: One-off usage, testing, or scripts outside a project context
See uv-commands.md for complete reference.
Quick Reference: Dependency Groups
[dependency-groups]
dev = ["ruff", "ty"]
test = ["pytest", "pytest-cov", "hypothesis"]
docs = ["sphinx", "myst-parser"]
Install with: uv sync --group dev --group test
Best Practices Checklist
- Use
src/layout for packages - Set
requires-python = ">=3.11" - Configure ruff with
select = ["ALL"]and explicit ignores - Use ty for type checking
- Enforce test coverage minimum (80%+)
- Use dependency groups instead of extras for dev tools
- Add
uv.lockto version control - Use PEP 723 for standalone scripts
Read Next
- migration-checklist.md - Step-by-step migration cleanup
- pyproject.md - Complete pyproject.toml reference
- uv-commands.md - uv command reference
- ruff-config.md - Ruff linting/formatting configuration
- testing.md - pytest and coverage setup
- pep723-scripts.md - PEP 723 inline script metadata
- prek.md - Fast pre-commit hooks with prek
- security-setup.md - Security hooks and dependency scanning
- dependabot.md - Automated dependency updates
Python Anti-Patterns Checklist
A reference checklist of common mistakes and anti-patterns in Python code. Review this before finalizing implementations to catch issues early.
When to Use This Skill
- Reviewing code before merge
- Debugging mysterious issues
- Teaching or learning Python best practices
- Establishing team coding standards
- Refactoring legacy code
Note: This skill focuses on what to avoid. For guidance on positive patterns and architecture, see the python-design-patterns skill.
Infrastructure Anti-Patterns
Scattered Timeout/Retry Logic
# BAD: Timeout logic duplicated everywhere
def fetch_user(user_id):
try:
return requests.get(url, timeout=30)
except Timeout:
logger.warning("Timeout fetching user")
return None
def fetch_orders(user_id):
try:
return requests.get(url, timeout=30)
except Timeout:
logger.warning("Timeout fetching orders")
return None
Fix: Centralize in decorators or client wrappers.
# GOOD: Centralized retry logic
@retry(stop=stop_after_attempt(3), wait=wait_exponential())
def http_get(url: str) -> Response:
return requests.get(url, timeout=30)
Double Retry
# BAD: Retrying at multiple layers
@retry(max_attempts=3) # Application retry
def call_service():
return client.request() # Client also has retry configured!
Fix: Retry at one layer only. Know your infrastructure's retry behavior.
Hard-Coded Configuration
# BAD: Secrets and config in code
DB_HOST = "prod-db.example.com"
API_KEY = "sk-12345"
def connect():
return psycopg.connect(f"host={DB_HOST}...")
Fix: Use environment variables with typed settings.
# GOOD
from pydantic_settings import BaseSettings
class Settings(BaseSettings):
db_host: str = Field(alias="DB_HOST")
api_key: str = Field(alias="API_KEY")
settings = Settings()
Architecture Anti-Patterns
Exposed Internal Types
# BAD: Leaking ORM model to API
@app.get("/users/{id}")
def get_user(id: str) -> UserModel: # SQLAlchemy model
return db.query(UserModel).get(id)
Fix: Use DTOs/response models.
# GOOD
@app.get("/users/{id}")
def get_user(id: str) -> UserResponse:
user = db.query(UserModel).get(id)
return UserResponse.from_orm(user)
Mixed I/O and Business Logic
# BAD: SQL embedded in business logic
def calculate_discount(user_id: str) -> float:
user = db.query("SELECT * FROM users WHERE id = ?", user_id)
orders = db.query("SELECT * FROM orders WHERE user_id = ?", user_id)
# Business logic mixed with data access
if len(orders) > 10:
return 0.15
return 0.0
Fix: Repository pattern. Keep business logic pure.
# GOOD
def calculate_discount(user: User, orders: list[Order]) -> float:
# Pure business logic, easily testable
if len(orders) > 10:
return 0.15
return 0.0
Error Handling Anti-Patterns
Bare Exception Handling
# BAD: Swallowing all exceptions
try:
process()
except Exception:
pass # Silent failure - bugs hidden forever
Fix: Catch specific exceptions. Log or handle appropriately.
# GOOD
try:
process()
except ConnectionError as e:
logger.warning("Connection failed, will retry", error=str(e))
raise
except ValueError as e:
logger.error("Invalid input", error=str(e))
raise BadRequestError(str(e))
Ignored Partial Failures
# BAD: Stops on first error
def process_batch(items):
results = []
for item in items:
result = process(item) # Raises on error - batch aborted
results.append(result)
return results
Fix: Capture both successes and failures.
# GOOD
def process_batch(items) -> BatchResult:
succeeded = {}
failed = {}
for idx, item in enumerate(items):
try:
succeeded[idx] = process(item)
except Exception as e:
failed[idx] = e
return BatchResult(succeeded, failed)
Missing Input Validation
# BAD: No validation
def create_user(data: dict):
return User(**data) # Crashes deep in code on bad input
Fix: Validate early at API boundaries.
# GOOD
def create_user(data: dict) -> User:
validated = CreateUserInput.model_validate(data)
return User.from_input(validated)
Resource Anti-Patterns
Unclosed Resources
# BAD: File never closed
def read_file(path):
f = open(path)
return f.read() # What if this raises?
Fix: Use context managers.
# GOOD
def read_file(path):
with open(path) as f:
return f.read()
Blocking in Async
# BAD: Blocks the entire event loop
async def fetch_data():
time.sleep(1) # Blocks everything!
response = requests.get(url) # Also blocks!
Fix: Use async-native libraries.
# GOOD
async def fetch_data():
await asyncio.sleep(1)
async with httpx.AsyncClient() as client:
response = await client.get(url)
Type Safety Anti-Patterns
Missing Type Hints
# BAD: No types
def process(data):
return data["value"] * 2
Fix: Annotate all public functions.
# GOOD
def process(data: dict[str, int]) -> int:
return data["value"] * 2
Untyped Collections
# BAD: Generic list without type parameter
def get_users() -> list:
...
Fix: Use type parameters.
# GOOD
def get_users() -> list[User]:
...
Testing Anti-Patterns
Only Testing Happy Paths
# BAD: Only tests success case
def test_create_user():
user = service.create_user(valid_data)
assert user.id is not None
Fix: Test error conditions and edge cases.
# GOOD
def test_create_user_success():
user = service.create_user(valid_data)
assert user.id is not None
def test_create_user_invalid_email():
with pytest.raises(ValueError, match="Invalid email"):
service.create_user(invalid_email_data)
def test_create_user_duplicate_email():
service.create_user(valid_data)
with pytest.raises(ConflictError):
service.create_user(valid_data)
Over-Mocking
# BAD: Mocking everything
def test_user_service():
mock_repo = Mock()
mock_cache = Mock()
mock_logger = Mock()
mock_metrics = Mock()
# Test doesn't verify real behavior
Fix: Use integration tests for critical paths. Mock only external services.
Quick Review Checklist
Before finalizing code, verify:
- No scattered timeout/retry logic (centralized)
- No double retry (app + infrastructure)
- No hard-coded configuration or secrets
- No exposed internal types (ORM models, protobufs)
- No mixed I/O and business logic
- No bare
except Exception: pass - No ignored partial failures in batches
- No missing input validation
- No unclosed resources (using context managers)
- No blocking calls in async code
- All public functions have type hints
- Collections have type parameters
- Error paths are tested
- Edge cases are covered
Common Fixes Summary
| Anti-Pattern | Fix |
|---|---|
| Scattered retry logic | Centralized decorators |
| Hard-coded config | Environment variables + pydantic-settings |
| Exposed ORM models | DTO/response schemas |
| Mixed I/O + logic | Repository pattern |
| Bare except | Catch specific exceptions |
| Batch stops on error | Return BatchResult with successes/failures |
| No validation | Validate at boundaries with Pydantic |
| Unclosed resources | Context managers |
| Blocking in async | Async-native libraries |
| Missing types | Type annotations on all public APIs |
| Only happy path tests | Test errors and edge cases |
Python Background Jobs & Task Queues
Decouple long-running or unreliable work from request/response cycles. Return immediately to the user while background workers handle the heavy lifting asynchronously.
When to Use This Skill
- Processing tasks that take longer than a few seconds
- Sending emails, notifications, or webhooks
- Generating reports or exporting data
- Processing uploads or media transformations
- Integrating with unreliable external services
- Building event-driven architectures
Core Concepts
1. Task Queue Pattern
API accepts request, enqueues a job, returns immediately with a job ID. Workers process jobs asynchronously.
2. Idempotency
Tasks may be retried on failure. Design for safe re-execution.
3. Job State Machine
Jobs transition through states: pending → running → succeeded/failed.
4. At-Least-Once Delivery
Most queues guarantee at-least-once delivery. Your code must handle duplicates.
Quick Start
This skill uses Celery for examples, a widely adopted task queue. Alternatives like RQ, Dramatiq, and cloud-native solutions (AWS SQS, GCP Tasks) are equally valid choices.
from celery import Celery
app = Celery("tasks", broker="redis://localhost:6379")
@app.task
def send_email(to: str, subject: str, body: str) -> None:
# This runs in a background worker
email_client.send(to, subject, body)
# In your API handler
send_email.delay("user@example.com", "Welcome!", "Thanks for signing up")
Fundamental Patterns
Pattern 1: Return Job ID Immediately
For operations exceeding a few seconds, return a job ID and process asynchronously.
from uuid import uuid4
from dataclasses import dataclass
from enum import Enum
from datetime import datetime
class JobStatus(Enum):
PENDING = "pending"
RUNNING = "running"
SUCCEEDED = "succeeded"
FAILED = "failed"
@dataclass
class Job:
id: str
status: JobStatus
created_at: datetime
started_at: datetime | None = None
completed_at: datetime | None = None
result: dict | None = None
error: str | None = None
# API endpoint
async def start_export(request: ExportRequest) -> JobResponse:
"""Start export job and return job ID."""
job_id = str(uuid4())
# Persist job record
await jobs_repo.create(Job(
id=job_id,
status=JobStatus.PENDING,
created_at=datetime.utcnow(),
))
# Enqueue task for background processing
await task_queue.enqueue(
"export_data",
job_id=job_id,
params=request.model_dump(),
)
# Return immediately with job ID
return JobResponse(
job_id=job_id,
status="pending",
poll_url=f"/jobs/{job_id}",
)
Pattern 2: Celery Task Configuration
Configure Celery tasks with proper retry and timeout settings.
from celery import Celery
app = Celery("tasks", broker="redis://localhost:6379")
# Global configuration
app.conf.update(
task_time_limit=3600, # Hard limit: 1 hour
task_soft_time_limit=3000, # Soft limit: 50 minutes
task_acks_late=True, # Acknowledge after completion
task_reject_on_worker_lost=True,
worker_prefetch_multiplier=1, # Don't prefetch too many tasks
)
@app.task(
bind=True,
max_retries=3,
default_retry_delay=60,
autoretry_for=(ConnectionError, TimeoutError),
)
def process_payment(self, payment_id: str) -> dict:
"""Process payment with automatic retry on transient errors."""
try:
result = payment_gateway.charge(payment_id)
return {"status": "success", "transaction_id": result.id}
except PaymentDeclinedError as e:
# Don't retry permanent failures
return {"status": "declined", "reason": str(e)}
except TransientError as e:
# Retry with exponential backoff
raise self.retry(exc=e, countdown=2 ** self.request.retries * 60)
Pattern 3: Make Tasks Idempotent
Workers may retry on crash or timeout. Design for safe re-execution.
@app.task(bind=True)
def process_order(self, order_id: str) -> None:
"""Process order idempotently."""
order = orders_repo.get(order_id)
# Already processed? Return early
if order.status == OrderStatus.COMPLETED:
logger.info("Order already processed", order_id=order_id)
return
# Already in progress? Check if we should continue
if order.status == OrderStatus.PROCESSING:
# Use idempotency key to avoid double-charging
pass
# Process with idempotency key
result = payment_provider.charge(
amount=order.total,
idempotency_key=f"order-{order_id}", # Critical!
)
orders_repo.update(order_id, status=OrderStatus.COMPLETED)
Idempotency Strategies:
- Check-before-write: Verify state before action
- Idempotency keys: Use unique tokens with external services
- Upsert patterns:
INSERT ... ON CONFLICT UPDATE - Deduplication window: Track processed IDs for N hours
Pattern 4: Job State Management
Persist job state transitions for visibility and debugging.
class JobRepository:
"""Repository for managing job state."""
async def create(self, job: Job) -> Job:
"""Create new job record."""
await self._db.execute(
"""INSERT INTO jobs (id, status, created_at)
VALUES ($1, $2, $3)""",
job.id, job.status.value, job.created_at,
)
return job
async def update_status(
self,
job_id: str,
status: JobStatus,
**fields,
) -> None:
"""Update job status with timestamp."""
updates = {"status": status.value, **fields}
if status == JobStatus.RUNNING:
updates["started_at"] = datetime.utcnow()
elif status in (JobStatus.SUCCEEDED, JobStatus.FAILED):
updates["completed_at"] = datetime.utcnow()
await self._db.execute(
"UPDATE jobs SET status = $1, ... WHERE id = $2",
updates, job_id,
)
logger.info(
"Job status updated",
job_id=job_id,
status=status.value,
)
Detailed worked examples and patterns
Detailed sections (starting with ## Advanced Patterns) live in references/details.md. Read that file when the navigation summary above is insufficient.
Best Practices Summary
- Return immediately - Don't block requests for long operations
- Persist job state - Enable status polling and debugging
- Make tasks idempotent - Safe to retry on any failure
- Use idempotency keys - For external service calls
- Set timeouts - Both soft and hard limits
- Implement DLQ - Capture permanently failed tasks
- Log transitions - Track job state changes
- Retry appropriately - Exponential backoff for transient errors
- Don't retry permanent failures - Validation errors, invalid credentials
- Monitor queue depth - Alert on backlog growth
Python Code Style & Documentation
Consistent code style and clear documentation make codebases maintainable and collaborative. This skill covers modern Python tooling, naming conventions, and documentation standards.
When to Use This Skill
- Setting up linting and formatting for a new project
- Writing or reviewing docstrings
- Establishing team coding standards
- Configuring ruff, mypy, or pyright
- Reviewing code for style consistency
- Creating project documentation
Core Concepts
1. Automated Formatting
Let tools handle formatting debates. Configure once, enforce automatically.
2. Consistent Naming
Follow PEP 8 conventions with meaningful, descriptive names.
3. Documentation as Code
Docstrings should be maintained alongside the code they describe.
4. Type Annotations
Modern Python code should include type hints for all public APIs.
Quick Start
# Install modern tooling
pip install ruff mypy
# Configure in pyproject.toml
[tool.ruff]
line-length = 120
target-version = "py312" # Adjust based on your project's minimum Python version
[tool.mypy]
strict = true
Fundamental Patterns
Pattern 1: Modern Python Tooling
Use ruff as an all-in-one linter and formatter. It replaces flake8, isort, and black with a single fast tool.
# pyproject.toml
[tool.ruff]
line-length = 120
target-version = "py312" # Adjust based on your project's minimum Python version
[tool.ruff.lint]
select = [
"E", # pycodestyle errors
"W", # pycodestyle warnings
"F", # pyflakes
"I", # isort
"B", # flake8-bugbear
"C4", # flake8-comprehensions
"UP", # pyupgrade
"SIM", # flake8-simplify
]
ignore = ["E501"] # Line length handled by formatter
[tool.ruff.format]
quote-style = "double"
indent-style = "space"
Run with:
ruff check --fix . # Lint and auto-fix
ruff format . # Format code
Pattern 2: Type Checking Configuration
Configure strict type checking for production code.
# pyproject.toml
[tool.mypy]
python_version = "3.12"
strict = true
warn_return_any = true
warn_unused_ignores = true
disallow_untyped_defs = true
disallow_incomplete_defs = true
[[tool.mypy.overrides]]
module = "tests.*"
disallow_untyped_defs = false
Alternative: Use pyright for faster checking.
[tool.pyright]
pythonVersion = "3.12"
typeCheckingMode = "strict"
Pattern 3: Naming Conventions
Follow PEP 8 with emphasis on clarity over brevity.
Files and Modules:
# Good: Descriptive snake_case
user_repository.py
order_processing.py
http_client.py
# Avoid: Abbreviations
usr_repo.py
ord_proc.py
http_cli.py
Classes and Functions:
# Classes: PascalCase
class UserRepository:
pass
class HTTPClientFactory: # Acronyms stay uppercase
pass
# Functions and variables: snake_case
def get_user_by_email(email: str) -> User | None:
retry_count = 3
max_connections = 100
Constants:
# Module-level constants: SCREAMING_SNAKE_CASE
MAX_RETRY_ATTEMPTS = 3
DEFAULT_TIMEOUT_SECONDS = 30
API_BASE_URL = "https://api.example.com"
Pattern 4: Import Organization
Group imports in a consistent order: standard library, third-party, local.
# Standard library
import os
from collections.abc import Callable
from typing import Any
# Third-party packages
import httpx
from pydantic import BaseModel
from sqlalchemy import Column
# Local imports
from myproject.models import User
from myproject.services import UserService
Use absolute imports exclusively:
# Preferred
from myproject.utils import retry_decorator
# Avoid relative imports
from ..utils import retry_decorator
Advanced Patterns
Pattern 5: Google-Style Docstrings
Write docstrings for all public classes, methods, and functions.
Simple Function:
def get_user(user_id: str) -> User:
"""Retrieve a user by their unique identifier."""
...
Complex Function:
def process_batch(
items: list[Item],
max_workers: int = 4,
on_progress: Callable[[int, int], None] | None = None,
) -> BatchResult:
"""Process items concurrently using a worker pool.
Processes each item in the batch using the configured number of
workers. Progress can be monitored via the optional callback.
Args:
items: The items to process. Must not be empty.
max_workers: Maximum concurrent workers. Defaults to 4.
on_progress: Optional callback receiving (completed, total) counts.
Returns:
BatchResult containing succeeded items and any failures with
their associated exceptions.
Raises:
ValueError: If items is empty.
ProcessingError: If the batch cannot be processed.
Example:
>>> result = process_batch(items, max_workers=8)
>>> print(f"Processed {len(result.succeeded)} items")
"""
...
Class Docstring:
class UserService:
"""Service for managing user operations.
Provides methods for creating, retrieving, updating, and
deleting users with proper validation and error handling.
Attributes:
repository: The data access layer for user persistence.
logger: Logger instance for operation tracking.
Example:
>>> service = UserService(repository, logger)
>>> user = service.create_user(CreateUserInput(...))
"""
def __init__(self, repository: UserRepository, logger: Logger) -> None:
"""Initialize the user service.
Args:
repository: Data access layer for users.
logger: Logger for tracking operations.
"""
self.repository = repository
self.logger = logger
Pattern 6: Line Length and Formatting
Set line length to 120 characters for modern displays while maintaining readability.
# Good: Readable line breaks
def create_user(
email: str,
name: str,
role: UserRole = UserRole.MEMBER,
notify: bool = True,
) -> User:
...
# Good: Chain method calls clearly
result = (
db.query(User)
.filter(User.active == True)
.order_by(User.created_at.desc())
.limit(10)
.all()
)
# Good: Format long strings
error_message = (
f"Failed to process user {user_id}: "
f"received status {response.status_code} "
f"with body {response.text[:100]}"
)
Pattern 7: Project Documentation
README Structure:
# Project Name
Brief description of what the project does.
## Installation
\`\`\`bash
pip install myproject
\`\`\`
## Quick Start
\`\`\`python
from myproject import Client
client = Client(api_key="...")
result = client.process(data)
\`\`\`
## Configuration
Document environment variables and configuration options.
## Development
\`\`\`bash
pip install -e ".[dev]"
pytest
\`\`\`
CHANGELOG Format (Keep a Changelog):
# Changelog
## [Unreleased]
### Added
- New feature X
### Changed
- Modified behavior of Y
### Fixed
- Bug in Z
Best Practices Summary
- Use ruff - Single tool for linting and formatting
- Enable strict mypy - Catch type errors before runtime
- 120 character lines - Modern standard for readability
- Descriptive names - Clarity over brevity
- Absolute imports - More maintainable than relative
- Google-style docstrings - Consistent, readable documentation
- Document public APIs - Every public function needs a docstring
- Keep docs updated - Treat documentation as code
- Automate in CI - Run linters on every commit
- Target Python 3.10+ - For new projects, Python 3.12+ is recommended for modern language features
Python Configuration Management
Externalize configuration from code using environment variables and typed settings. Well-managed configuration enables the same code to run in any environment without modification.
When to Use This Skill
- Setting up a new project's configuration system
- Migrating from hardcoded values to environment variables
- Implementing pydantic-settings for typed configuration
- Managing secrets and sensitive values
- Creating environment-specific settings (dev/staging/prod)
- Validating configuration at application startup
Core Concepts
1. Externalized Configuration
All environment-specific values (URLs, secrets, feature flags) come from environment variables, not code.
2. Typed Settings
Parse and validate configuration into typed objects at startup, not scattered throughout code.
3. Fail Fast
Validate all required configuration at application boot. Missing config should crash immediately with a clear message.
4. Sensible Defaults
Provide reasonable defaults for local development while requiring explicit values for sensitive settings.
Quick Start
from pydantic_settings import BaseSettings
from pydantic import Field
class Settings(BaseSettings):
database_url: str = Field(alias="DATABASE_URL")
api_key: str = Field(alias="API_KEY")
debug: bool = Field(default=False, alias="DEBUG")
settings = Settings() # Loads from environment
Fundamental Patterns
Pattern 1: Typed Settings with Pydantic
Create a central settings class that loads and validates all configuration.
from pydantic_settings import BaseSettings
from pydantic import Field, PostgresDsn, ValidationError
import sys
class Settings(BaseSettings):
"""Application configuration loaded from environment variables."""
# Database
db_host: str = Field(alias="DB_HOST")
db_port: int = Field(default=5432, alias="DB_PORT")
db_name: str = Field(alias="DB_NAME")
db_user: str = Field(alias="DB_USER")
db_password: str = Field(alias="DB_PASSWORD")
# Redis
redis_url: str = Field(default="redis://localhost:6379", alias="REDIS_URL")
# API Keys
api_secret_key: str = Field(alias="API_SECRET_KEY")
# Feature flags
enable_new_feature: bool = Field(default=False, alias="ENABLE_NEW_FEATURE")
model_config = {
"env_file": ".env",
"env_file_encoding": "utf-8",
}
# Create singleton instance at module load
try:
settings = Settings()
except ValidationError as e:
print(f"Configuration error:\n{e}")
sys.exit(1)
Import settings throughout your application:
from myapp.config import settings
def get_database_connection():
return connect(
host=settings.db_host,
port=settings.db_port,
database=settings.db_name,
)
Pattern 2: Fail Fast on Missing Configuration
Required settings should crash the application immediately with a clear error.
from pydantic_settings import BaseSettings
from pydantic import Field, ValidationError
import sys
class Settings(BaseSettings):
# Required - no default means it must be set
api_key: str = Field(alias="API_KEY")
database_url: str = Field(alias="DATABASE_URL")
# Optional with defaults
log_level: str = Field(default="INFO", alias="LOG_LEVEL")
try:
settings = Settings()
except ValidationError as e:
print("=" * 60)
print("CONFIGURATION ERROR")
print("=" * 60)
for error in e.errors():
field = error["loc"][0]
print(f" - {field}: {error['msg']}")
print("\nPlease set the required environment variables.")
sys.exit(1)
A clear error at startup is better than a cryptic None failure mid-request.
Pattern 3: Local Development Defaults
Provide sensible defaults for local development while requiring explicit values for secrets.
class Settings(BaseSettings):
# Has local default, but prod will override
db_host: str = Field(default="localhost", alias="DB_HOST")
db_port: int = Field(default=5432, alias="DB_PORT")
# Always required - no default for secrets
db_password: str = Field(alias="DB_PASSWORD")
api_secret_key: str = Field(alias="API_SECRET_KEY")
# Development convenience
debug: bool = Field(default=False, alias="DEBUG")
model_config = {"env_file": ".env"}
Create a .env file for local development (never commit this):
# .env (add to .gitignore)
DB_PASSWORD=local_dev_password
API_SECRET_KEY=dev-secret-key
DEBUG=true
Pattern 4: Namespaced Environment Variables
Prefix related variables for clarity and easy debugging.
# Database configuration
DB_HOST=localhost
DB_PORT=5432
DB_NAME=myapp
DB_USER=admin
DB_PASSWORD=secret
# Redis configuration
REDIS_URL=redis://localhost:6379
REDIS_MAX_CONNECTIONS=10
# Authentication
AUTH_SECRET_KEY=your-secret-key
AUTH_TOKEN_EXPIRY_SECONDS=3600
AUTH_ALGORITHM=HS256
# Feature flags
FEATURE_NEW_CHECKOUT=true
FEATURE_BETA_UI=false
Makes the database settings easy to list for debugging. Print only the names, so no value (such as a password inside a connection URL) reaches your terminal or logs:
env | grep -o '^DB_[A-Za-z0-9_]*'
Detailed worked examples and patterns
Detailed sections (starting with ## Advanced Patterns) live in references/details.md. Read that file when the navigation summary above is insufficient.
Best Practices Summary
- Never hardcode config - All environment-specific values from env vars
- Use typed settings - Pydantic-settings with validation
- Fail fast - Crash on missing required config at startup
- Provide dev defaults - Make local development easy
- Never commit secrets - Use
.envfiles (gitignored) or secret managers - Namespace variables -
DB_HOST,REDIS_URLfor clarity - Import settings singleton - Don't call
os.getenv()throughout code - Document all variables - README should list required env vars
- Validate early - Check config correctness at boot time
- Use secrets_dir - Support mounted secrets in containers
Python Design Patterns
Write maintainable Python code using fundamental design principles. These patterns help you build systems that are easy to understand, test, and modify.
When to Use This Skill
- Designing new components or services
- Refactoring complex or tangled code
- Deciding whether to create an abstraction
- Choosing between inheritance and composition
- Evaluating code complexity and coupling
- Planning modular architectures
Core Concepts
1. KISS (Keep It Simple)
Choose the simplest solution that works. Complexity must be justified by concrete requirements.
2. Single Responsibility (SRP)
Each unit should have one reason to change. Separate concerns into focused components.
3. Composition Over Inheritance
Build behavior by combining objects, not extending classes.
4. Rule of Three
Wait until you have three instances before abstracting. Duplication is often better than premature abstraction.
Quick Start
# Simple beats clever
# Instead of a factory/registry pattern:
FORMATTERS = {"json": JsonFormatter, "csv": CsvFormatter}
def get_formatter(name: str) -> Formatter:
return FORMATTERS[name]()
Detailed patterns and worked examples
Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.
Best Practices Summary
- Keep it simple - Choose the simplest solution that works
- Single responsibility - Each unit has one reason to change
- Separate concerns - Distinct layers with clear purposes
- Compose, don't inherit - Combine objects for flexibility
- Rule of three - Wait before abstracting
- Keep functions small - 20-50 lines (varies by complexity), one purpose
- Inject dependencies - Constructor injection for testability
- Delete before abstracting - Remove dead code, then consider patterns
- Test each layer - Isolated tests for each concern
- Explicit over clever - Readable code beats elegant code
Troubleshooting
A class is growing and seems to have multiple responsibilities, but splitting it feels wrong. Apply the "reason to change" test: list every change that could require editing this class. If the list has items from different domains (e.g., HTTP parsing AND business rules AND formatting), split it. If all changes stem from the same domain concern, the class may be appropriately sized.
Injecting all dependencies through the constructor is producing constructors with 7+ parameters. This is a sign of too many responsibilities in one class, not a problem with dependency injection. Split the class into smaller units first, then each constructor naturally becomes smaller.
Composition is producing deeply nested wrapper objects that are hard to trace. Keep the composition shallow (2-3 levels). If wrapping is the only mechanism, consider whether a Protocol-based approach or simple function composition would be cleaner than a chain of decorator objects.
The rule of three says not to abstract yet, but the duplication is causing bugs when one copy is updated but not the other. Duplication that diverges in dangerous ways should be abstracted sooner. The rule of three is a heuristic, not a law. If the copies are already diverging incorrectly, extract immediately and add a test that exercises the shared behavior.
A service layer is importing from the API layer, breaking the dependency direction. This is a layering violation. The service layer must not import from handlers. Introduce a shared types/models layer that both can import from, keeping the dependency arrow pointing downward (API → Service → Repository).
Related Skills
- python-testing-patterns — Test each layer in isolation using the dependency injection structure established here
- python-project-structure — Organize modules and directory layout so layer boundaries are explicit from the start
Python Error Handling
Build robust Python applications with proper input validation, meaningful exceptions, and graceful failure handling. Good error handling makes debugging easier and systems more reliable.
When to Use This Skill
- Validating user input and API parameters
- Designing exception hierarchies for applications
- Handling partial failures in batch operations
- Converting external data to domain types
- Building user-friendly error messages
- Implementing fail-fast validation patterns
Core Concepts
1. Fail Fast
Validate inputs early, before expensive operations. Report all validation errors at once when possible.
2. Meaningful Exceptions
Use appropriate exception types with context. Messages should explain what failed, why, and how to fix it.
3. Partial Failures
In batch operations, don't let one failure abort everything. Track successes and failures separately.
4. Preserve Context
Chain exceptions to maintain the full error trail for debugging.
Quick Start
def fetch_page(url: str, page_size: int) -> Page:
if not url:
raise ValueError("'url' is required")
if not 1 <= page_size <= 100:
raise ValueError(f"'page_size' must be 1-100, got {page_size}")
# Now safe to proceed...
Fundamental Patterns
Pattern 1: Early Input Validation
Validate all inputs at API boundaries before any processing begins.
def process_order(
order_id: str,
quantity: int,
discount_percent: float,
) -> OrderResult:
"""Process an order with validation."""
# Validate required fields
if not order_id:
raise ValueError("'order_id' is required")
# Validate ranges
if quantity <= 0:
raise ValueError(f"'quantity' must be positive, got {quantity}")
if not 0 <= discount_percent <= 100:
raise ValueError(
f"'discount_percent' must be 0-100, got {discount_percent}"
)
# Validation passed, proceed with processing
return _process_validated_order(order_id, quantity, discount_percent)
Pattern 2: Convert to Domain Types Early
Parse strings and external data into typed domain objects at system boundaries.
from enum import Enum
class OutputFormat(Enum):
JSON = "json"
CSV = "csv"
PARQUET = "parquet"
def parse_output_format(value: str) -> OutputFormat:
"""Parse string to OutputFormat enum.
Args:
value: Format string from user input.
Returns:
Validated OutputFormat enum member.
Raises:
ValueError: If format is not recognized.
"""
try:
return OutputFormat(value.lower())
except ValueError:
valid_formats = [f.value for f in OutputFormat]
raise ValueError(
f"Invalid format '{value}'. "
f"Valid options: {', '.join(valid_formats)}"
)
# Usage at API boundary
def export_data(data: list[dict], format_str: str) -> bytes:
output_format = parse_output_format(format_str) # Fail fast
# Rest of function uses typed OutputFormat
...
Pattern 3: Pydantic for Complex Validation
Use Pydantic models for structured input validation with automatic error messages.
from pydantic import BaseModel, Field, field_validator
class CreateUserInput(BaseModel):
"""Input model for user creation."""
email: str = Field(..., min_length=5, max_length=255)
name: str = Field(..., min_length=1, max_length=100)
age: int = Field(ge=0, le=150)
@field_validator("email")
@classmethod
def validate_email_format(cls, v: str) -> str:
if "@" not in v or "." not in v.split("@")[-1]:
raise ValueError("Invalid email format")
return v.lower()
@field_validator("name")
@classmethod
def normalize_name(cls, v: str) -> str:
return v.strip().title()
# Usage
try:
user_input = CreateUserInput(
email="user@example.com",
name="john doe",
age=25,
)
except ValidationError as e:
# Pydantic provides detailed error information
print(e.errors())
Pattern 4: Map Errors to Standard Exceptions
Use Python's built-in exception types appropriately, adding context as needed.
| Failure Type | Exception | Example |
|---|---|---|
| Invalid input | ValueError | Bad parameter values |
| Wrong type | TypeError | Expected string, got int |
| Missing item | KeyError | Dict key not found |
| Operational failure | RuntimeError | Service unavailable |
| Timeout | TimeoutError | Operation took too long |
| File not found | FileNotFoundError | Path doesn't exist |
| Permission denied | PermissionError | Access forbidden |
# Good: Specific exception with context
raise ValueError(f"'page_size' must be 1-100, got {page_size}")
# Avoid: Generic exception, no context
raise Exception("Invalid parameter")
Detailed worked examples and patterns
Detailed sections (starting with ## Advanced Patterns) live in references/details.md. Read that file when the navigation summary above is insufficient.
Best Practices Summary
- Validate early - Check inputs before expensive operations
- Use specific exceptions -
ValueError,TypeError, not genericException - Include context - Messages should explain what, why, and how to fix
- Convert types at boundaries - Parse strings to enums/domain types early
- Chain exceptions - Use
raise ... from eto preserve debug info - Handle partial failures - Don't abort batches on single item errors
- Use Pydantic - For complex input validation with structured errors
- Document failure modes - Docstrings should list possible exceptions
- Log with context - Include IDs, counts, and other debugging info
- Test error paths - Verify exceptions are raised correctly
Python Observability
Instrument Python applications with structured logs, metrics, and traces. When something breaks in production, you need to answer "what, where, and why" without deploying new code.
When to Use This Skill
- Adding structured logging to applications
- Implementing metrics collection with Prometheus
- Setting up distributed tracing across services
- Propagating correlation IDs through request chains
- Debugging production issues
- Building observability dashboards
Core Concepts
1. Structured Logging
Emit logs as JSON with consistent fields for production environments. Machine-readable logs enable powerful queries and alerts. For local development, consider human-readable formats.
2. The Four Golden Signals
Track latency, traffic, errors, and saturation for every service boundary.
3. Correlation IDs
Thread a unique ID through all logs and spans for a single request, enabling end-to-end tracing.
4. Bounded Cardinality
Keep metric label values bounded. Unbounded labels (like user IDs) explode storage costs.
Quick Start
import structlog
structlog.configure(
processors=[
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.JSONRenderer(),
],
)
logger = structlog.get_logger()
logger.info("Request processed", user_id="123", duration_ms=45)
Fundamental Patterns
Pattern 1: Structured Logging with Structlog
Configure structlog for JSON output with consistent fields.
import logging
import structlog
def configure_logging(log_level: str = "INFO") -> None:
"""Configure structured logging for the application."""
structlog.configure(
processors=[
structlog.contextvars.merge_contextvars,
structlog.processors.add_log_level,
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.StackInfoRenderer(),
structlog.processors.format_exc_info,
structlog.processors.JSONRenderer(),
],
wrapper_class=structlog.make_filtering_bound_logger(
getattr(logging, log_level.upper())
),
context_class=dict,
logger_factory=structlog.PrintLoggerFactory(),
cache_logger_on_first_use=True,
)
# Initialize at application startup
configure_logging("INFO")
logger = structlog.get_logger()
Pattern 2: Consistent Log Fields
Every log entry should include standard fields for filtering and correlation.
import structlog
from contextvars import ContextVar
# Store correlation ID in context
correlation_id: ContextVar[str] = ContextVar("correlation_id", default="")
logger = structlog.get_logger()
def process_request(request: Request) -> Response:
"""Process request with structured logging."""
logger.info(
"Request received",
correlation_id=correlation_id.get(),
method=request.method,
path=request.path,
user_id=request.user_id,
)
try:
result = handle_request(request)
logger.info(
"Request completed",
correlation_id=correlation_id.get(),
status_code=200,
duration_ms=elapsed,
)
return result
except Exception as e:
logger.error(
"Request failed",
correlation_id=correlation_id.get(),
error_type=type(e).__name__,
error_message=str(e),
)
raise
Pattern 3: Semantic Log Levels
Use log levels consistently across the application.
| Level | Purpose | Examples |
|---|---|---|
DEBUG | Development diagnostics | Variable values, internal state |
INFO | Request lifecycle, operations | Request start/end, job completion |
WARNING | Recoverable anomalies | Retry attempts, fallback used |
ERROR | Failures needing attention | Exceptions, service unavailable |
# DEBUG: Detailed internal information
logger.debug("Cache lookup", key=cache_key, hit=cache_hit)
# INFO: Normal operational events
logger.info("Order created", order_id=order.id, total=order.total)
# WARNING: Abnormal but handled situations
logger.warning(
"Rate limit approaching",
current_rate=950,
limit=1000,
reset_seconds=30,
)
# ERROR: Failures requiring investigation
logger.error(
"Payment processing failed",
order_id=order.id,
error=str(e),
payment_provider="stripe",
)
Never log expected behavior at ERROR. A user entering a wrong password is INFO, not ERROR.
Pattern 4: Correlation ID Propagation
Generate a unique ID at ingress and thread it through all operations.
from contextvars import ContextVar
import uuid
import structlog
correlation_id: ContextVar[str] = ContextVar("correlation_id", default="")
def set_correlation_id(cid: str | None = None) -> str:
"""Set correlation ID for current context."""
cid = cid or str(uuid.uuid4())
correlation_id.set(cid)
structlog.contextvars.bind_contextvars(correlation_id=cid)
return cid
# FastAPI middleware example
from fastapi import Request
async def correlation_middleware(request: Request, call_next):
"""Middleware to set and propagate correlation ID."""
# Use incoming header or generate new
cid = request.headers.get("X-Correlation-ID") or str(uuid.uuid4())
set_correlation_id(cid)
response = await call_next(request)
response.headers["X-Correlation-ID"] = cid
return response
Propagate to outbound requests:
import httpx
async def call_downstream_service(endpoint: str, data: dict) -> dict:
"""Call downstream service with correlation ID."""
async with httpx.AsyncClient() as client:
response = await client.post(
endpoint,
json=data,
headers={"X-Correlation-ID": correlation_id.get()},
)
return response.json()
Detailed worked examples and patterns
Detailed sections (starting with ## Advanced Patterns) live in references/details.md. Read that file when the navigation summary above is insufficient.
Best Practices Summary
- Use structured logging - JSON logs with consistent fields
- Propagate correlation IDs - Thread through all requests and logs
- Track the four golden signals - Latency, traffic, errors, saturation
- Bound label cardinality - Never use unbounded values as metric labels
- Log at appropriate levels - Don't cry wolf with ERROR
- Include context - User ID, request ID, operation name in logs
- Use context managers - Consistent timing and error handling
- Separate concerns - Observability code shouldn't pollute business logic
- Test your observability - Verify logs and metrics in integration tests
- Set up alerts - Metrics are useless without alerting
Python Packaging
Comprehensive guide to creating, structuring, and distributing Python packages using modern packaging tools, pyproject.toml, and publishing to PyPI.
When to Use This Skill
- Creating Python libraries for distribution
- Building command-line tools with entry points
- Publishing packages to PyPI or private repositories
- Setting up Python project structure
- Creating installable packages with dependencies
- Building wheels and source distributions
- Versioning and releasing Python packages
- Creating namespace packages
- Implementing package metadata and classifiers
Core Concepts
1. Package Structure
- Source layout:
src/package_name/(recommended) - Flat layout:
package_name/(simpler but less flexible) - Package metadata: pyproject.toml, setup.py, or setup.cfg
- Distribution formats: wheel (.whl) and source distribution (.tar.gz)
2. Modern Packaging Standards
- PEP 517/518: Build system requirements
- PEP 621: Metadata in pyproject.toml
- PEP 660: Editable installs
- pyproject.toml: Single source of configuration
3. Build Backends
- setuptools: Traditional, widely used
- hatchling: Modern, opinionated
- flit: Lightweight, for pure Python
- poetry: Dependency management + packaging
4. Distribution
- PyPI: Python Package Index (public)
- TestPyPI: Testing before production
- Private repositories: JFrog, AWS CodeArtifact, etc.
Quick Start
Minimal Package Structure
my-package/
├── pyproject.toml
├── README.md
├── LICENSE
├── src/
│ └── my_package/
│ ├── __init__.py
│ └── module.py
└── tests/
└── test_module.py
Minimal pyproject.toml
[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"
[project]
name = "my-package"
version = "0.1.0"
description = "A short description"
authors = [{name = "Your Name", email = "you@example.com"}]
readme = "README.md"
requires-python = ">=3.8"
dependencies = [
"requests>=2.28.0",
]
[project.optional-dependencies]
dev = [
"pytest>=7.0",
"black>=22.0",
]
Package Structure Patterns
Pattern 1: Source Layout (Recommended)
my-package/
├── pyproject.toml
├── README.md
├── LICENSE
├── .gitignore
├── src/
│ └── my_package/
│ ├── __init__.py
│ ├── core.py
│ ├── utils.py
│ └── py.typed # For type hints
├── tests/
│ ├── __init__.py
│ ├── test_core.py
│ └── test_utils.py
└── docs/
└── index.md
Advantages:
- Prevents accidentally importing from source
- Cleaner test imports
- Better isolation
pyproject.toml for source layout:
[tool.setuptools.packages.find]
where = ["src"]
Pattern 2: Flat Layout
my-package/
├── pyproject.toml
├── README.md
├── my_package/
│ ├── __init__.py
│ └── module.py
└── tests/
└── test_module.py
Simpler but:
- Can import package without installing
- Less professional for libraries
Pattern 3: Multi-Package Project
project/
├── pyproject.toml
├── packages/
│ ├── package-a/
│ │ └── src/
│ │ └── package_a/
│ └── package-b/
│ └── src/
│ └── package_b/
└── tests/
Detailed patterns and worked examples
Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.
Python Development Patterns
Idiomatic Python patterns and best practices for building robust, efficient, and maintainable applications.
When to Activate
- Writing new Python code
- Reviewing Python code
- Refactoring existing Python code
- Designing Python packages/modules
Core Principles
1. Readability Counts
Python prioritizes readability. Code should be obvious and easy to understand.
# Good: Clear and readable
def get_active_users(users: list[User]) -> list[User]:
"""Return only active users from the provided list."""
return [user for user in users if user.is_active]
# Bad: Clever but confusing
def get_active_users(u):
return [x for x in u if x.a]
2. Explicit is Better Than Implicit
Avoid magic; be clear about what your code does.
# Good: Explicit configuration
import logging
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
# Bad: Hidden side effects
import some_module
some_module.setup() # What does this do?
3. EAFP - Easier to Ask Forgiveness Than Permission
Python prefers exception handling over checking conditions.
# Good: EAFP style
def get_value(dictionary: dict, key: str, default_value: Any = None) -> Any:
try:
return dictionary[key]
except KeyError:
return default_value
# Bad: LBYL (Look Before You Leap) style
def get_value(dictionary: dict, key: str, default_value: Any = None) -> Any:
if key in dictionary:
return dictionary[key]
else:
return default_value
Type Hints
Basic Type Annotations
from typing import Optional, List, Dict, Any
def process_user(
user_id: str,
data: Dict[str, Any],
active: bool = True
) -> Optional[User]:
"""Process a user and return the updated User or None."""
if not active:
return None
return User(user_id, data)
Modern Type Hints (Python 3.9+)
# Python 3.9+ - Use built-in types
def process_items(items: list[str]) -> dict[str, int]:
return {item: len(item) for item in items}
# Python 3.8 and earlier - Use typing module
from typing import List, Dict
def process_items(items: List[str]) -> Dict[str, int]:
return {item: len(item) for item in items}
Type Aliases and TypeVar
from typing import TypeVar, Union
# Type alias for complex types
JSON = Union[dict[str, Any], list[Any], str, int, float, bool, None]
def parse_json(data: str) -> JSON:
return json.loads(data)
# Generic types
T = TypeVar('T')
def first(items: list[T]) -> T | None:
"""Return the first item or None if list is empty."""
return items[0] if items else None
Protocol-Based Duck Typing
from typing import Protocol
class Renderable(Protocol):
def render(self) -> str:
"""Render the object to a string."""
def render_all(items: list[Renderable]) -> str:
"""Render all items that implement the Renderable protocol."""
return "\n".join(item.render() for item in items)
Error Handling Patterns
Specific Exception Handling
# Good: Catch specific exceptions
def load_config(path: str) -> Config:
try:
with open(path) as f:
return Config.from_json(f.read())
except FileNotFoundError as e:
raise ConfigError(f"Config file not found: {path}") from e
except json.JSONDecodeError as e:
raise ConfigError(f"Invalid JSON in config: {path}") from e
# Bad: Bare except
def load_config(path: str) -> Config:
try:
with open(path) as f:
return Config.from_json(f.read())
except:
return None # Silent failure!
Exception Chaining
def process_data(data: str) -> Result:
try:
parsed = json.loads(data)
except json.JSONDecodeError as e:
# Chain exceptions to preserve the traceback
raise ValueError(f"Failed to parse data: {data}") from e
Custom Exception Hierarchy
class AppError(Exception):
"""Base exception for all application errors."""
pass
class ValidationError(AppError):
"""Raised when input validation fails."""
pass
class NotFoundError(AppError):
"""Raised when a requested resource is not found."""
pass
# Usage
def get_user(user_id: str) -> User:
user = db.find_user(user_id)
if not user:
raise NotFoundError(f"User not found: {user_id}")
return user
Context Managers
Resource Management
# Good: Using context managers
def process_file(path: str) -> str:
with open(path, 'r') as f:
return f.read()
# Bad: Manual resource management
def process_file(path: str) -> str:
f = open(path, 'r')
try:
return f.read()
finally:
f.close()
Custom Context Managers
from contextlib import contextmanager
@contextmanager
def timer(name: str):
"""Context manager to time a block of code."""
start = time.perf_counter()
yield
elapsed = time.perf_counter() - start
print(f"{name} took {elapsed:.4f} seconds")
# Usage
with timer("data processing"):
process_large_dataset()
Context Manager Classes
class DatabaseTransaction:
def __init__(self, connection):
self.connection = connection
def __enter__(self):
self.connection.begin_transaction()
return self
def __exit__(self, exc_type, exc_val, exc_tb):
if exc_type is None:
self.connection.commit()
else:
self.connection.rollback()
return False # Don't suppress exceptions
# Usage
with DatabaseTransaction(conn):
user = conn.create_user(user_data)
conn.create_profile(user.id, profile_data)
Comprehensions and Generators
List Comprehensions
# Good: List comprehension for simple transformations
names = [user.name for user in users if user.is_active]
# Bad: Manual loop
names = []
for user in users:
if user.is_active:
names.append(user.name)
# Complex comprehensions should be expanded
# Bad: Too complex
result = [x * 2 for x in items if x > 0 if x % 2 == 0]
# Good: Use a generator function
def filter_and_transform(items: Iterable[int]) -> list[int]:
result = []
for x in items:
if x > 0 and x % 2 == 0:
result.append(x * 2)
return result
Generator Expressions
# Good: Generator for lazy evaluation
total = sum(x * x for x in range(1_000_000))
# Bad: Creates large intermediate list
total = sum([x * x for x in range(1_000_000)])
Generator Functions
def read_large_file(path: str) -> Iterator[str]:
"""Read a large file line by line."""
with open(path) as f:
for line in f:
yield line.strip()
# Usage
for line in read_large_file("huge.txt"):
process(line)
Data Classes and Named Tuples
Data Classes
from dataclasses import dataclass, field
from datetime import datetime
@dataclass
class User:
"""User entity with automatic __init__, __repr__, and __eq__."""
id: str
name: str
email: str
created_at: datetime = field(default_factory=datetime.now)
is_active: bool = True
# Usage
user = User(
id="123",
name="Alice",
email="alice@example.com"
)
Data Classes with Validation
@dataclass
class User:
email: str
age: int
def __post_init__(self):
# Validate email format
if "@" not in self.email:
raise ValueError(f"Invalid email: {self.email}")
# Validate age range
if self.age < 0 or self.age > 150:
raise ValueError(f"Invalid age: {self.age}")
Named Tuples
from typing import NamedTuple
class Point(NamedTuple):
"""Immutable 2D point."""
x: float
y: float
def distance(self, other: 'Point') -> float:
return ((self.x - other.x) ** 2 + (self.y - other.y) ** 2) ** 0.5
# Usage
p1 = Point(0, 0)
p2 = Point(3, 4)
print(p1.distance(p2)) # 5.0
Decorators
Function Decorators
import functools
import time
def timer(func: Callable) -> Callable:
"""Decorator to time function execution."""
@functools.wraps(func)
def wrapper(*args, **kwargs):
start = time.perf_counter()
result = func(*args, **kwargs)
elapsed = time.perf_counter() - start
print(f"{func.__name__} took {elapsed:.4f}s")
return result
return wrapper
@timer
def slow_function():
time.sleep(1)
# slow_function() prints: slow_function took 1.0012s
Parameterized Decorators
def repeat(times: int):
"""Decorator to repeat a function multiple times."""
def decorator(func: Callable) -> Callable:
@functools.wraps(func)
def wrapper(*args, **kwargs):
results = []
for _ in range(times):
results.append(func(*args, **kwargs))
return results
return wrapper
return decorator
@repeat(times=3)
def greet(name: str) -> str:
return f"Hello, {name}!"
# greet("Alice") returns ["Hello, Alice!", "Hello, Alice!", "Hello, Alice!"]
Class-Based Decorators
class CountCalls:
"""Decorator that counts how many times a function is called."""
def __init__(self, func: Callable):
functools.update_wrapper(self, func)
self.func = func
self.count = 0
def __call__(self, *args, **kwargs):
self.count += 1
print(f"{self.func.__name__} has been called {self.count} times")
return self.func(*args, **kwargs)
@CountCalls
def process():
pass
# Each call to process() prints the call count
Concurrency Patterns
Threading for I/O-Bound Tasks
import concurrent.futures
import threading
def fetch_url(url: str) -> str:
"""Fetch a URL (I/O-bound operation)."""
import urllib.request
with urllib.request.urlopen(url) as response:
return response.read().decode()
def fetch_all_urls(urls: list[str]) -> dict[str, str]:
"""Fetch multiple URLs concurrently using threads."""
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
future_to_url = {executor.submit(fetch_url, url): url for url in urls}
results = {}
for future in concurrent.futures.as_completed(future_to_url):
url = future_to_url[future]
try:
results[url] = future.result()
except Exception as e:
results[url] = f"Error: {e}"
return results
Multiprocessing for CPU-Bound Tasks
def process_data(data: list[int]) -> int:
"""CPU-intensive computation."""
return sum(x ** 2 for x in data)
def process_all(datasets: list[list[int]]) -> list[int]:
"""Process multiple datasets using multiple processes."""
with concurrent.futures.ProcessPoolExecutor() as executor:
results = list(executor.map(process_data, datasets))
return results
Async/Await for Concurrent I/O
import asyncio
async def fetch_async(url: str) -> str:
"""Fetch a URL asynchronously."""
import aiohttp
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
return await response.text()
async def fetch_all(urls: list[str]) -> dict[str, str]:
"""Fetch multiple URLs concurrently."""
tasks = [fetch_async(url) for url in urls]
results = await asyncio.gather(*tasks, return_exceptions=True)
return dict(zip(urls, results))
Package Organization
Standard Project Layout
myproject/
├── src/
│ └── mypackage/
│ ├── __init__.py
│ ├── main.py
│ ├── api/
│ │ ├── __init__.py
│ │ └── routes.py
│ ├── models/
│ │ ├── __init__.py
│ │ └── user.py
│ └── utils/
│ ├── __init__.py
│ └── helpers.py
├── tests/
│ ├── __init__.py
│ ├── conftest.py
│ ├── test_api.py
│ └── test_models.py
├── pyproject.toml
├── README.md
└── .gitignore
Import Conventions
# Good: Import order - stdlib, third-party, local
import os
import sys
from pathlib import Path
import requests
from fastapi import FastAPI
from mypackage.models import User
from mypackage.utils import format_name
# Good: Use isort for automatic import sorting
# pip install isort
init.py for Package Exports
# mypackage/__init__.py
"""mypackage - A sample Python package."""
__version__ = "1.0.0"
# Export main classes/functions at package level
from mypackage.models import User, Post
from mypackage.utils import format_name
__all__ = ["User", "Post", "format_name"]
Memory and Performance
Using slots for Memory Efficiency
# Bad: Regular class uses __dict__ (more memory)
class Point:
def __init__(self, x: float, y: float):
self.x = x
self.y = y
# Good: __slots__ reduces memory usage
class Point:
__slots__ = ['x', 'y']
def __init__(self, x: float, y: float):
self.x = x
self.y = y
Generator for Large Data
# Bad: Returns full list in memory
def read_lines(path: str) -> list[str]:
with open(path) as f:
return [line.strip() for line in f]
# Good: Yields lines one at a time
def read_lines(path: str) -> Iterator[str]:
with open(path) as f:
for line in f:
yield line.strip()
Avoid String Concatenation in Loops
# Bad: O(n²) due to string immutability
result = ""
for item in items:
result += str(item)
# Good: O(n) using join
result = "".join(str(item) for item in items)
# Good: Using StringIO for building
from io import StringIO
buffer = StringIO()
for item in items:
buffer.write(str(item))
result = buffer.getvalue()
Python Tooling Integration
Essential Commands
# Code formatting
black .
isort .
# Linting
ruff check .
pylint mypackage/
# Type checking
mypy .
# Testing
pytest --cov=mypackage --cov-report=html
# Security scanning
bandit -r .
# Dependency management
pip-audit
safety check
pyproject.toml Configuration
[project]
name = "mypackage"
version = "1.0.0"
requires-python = ">=3.9"
dependencies = [
"requests>=2.31.0",
"pydantic>=2.0.0",
]
[project.optional-dependencies]
dev = [
"pytest>=7.4.0",
"pytest-cov>=4.1.0",
"black>=23.0.0",
"ruff>=0.1.0",
"mypy>=1.5.0",
]
[tool.black]
line-length = 88
target-version = ['py39']
[tool.ruff]
line-length = 88
select = ["E", "F", "I", "N", "W"]
[tool.mypy]
python_version = "3.9"
warn_return_any = true
warn_unused_configs = true
disallow_untyped_defs = true
[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "--cov=mypackage --cov-report=term-missing"
Quick Reference: Python Idioms
| Idiom | Description |
|---|---|
| EAFP | Easier to Ask Forgiveness than Permission |
| Context managers | Use with for resource management |
| List comprehensions | For simple transformations |
| Generators | For lazy evaluation and large datasets |
| Type hints | Annotate function signatures |
| Dataclasses | For data containers with auto-generated methods |
__slots__ | For memory optimization |
| f-strings | For string formatting (Python 3.6+) |
pathlib.Path | For path operations (Python 3.4+) |
enumerate | For index-element pairs in loops |
Anti-Patterns to Avoid
# Bad: Mutable default arguments
def append_to(item, items=[]):
items.append(item)
return items
# Good: Use None and create new list
def append_to(item, items=None):
if items is None:
items = []
items.append(item)
return items
# Bad: Checking type with type()
if type(obj) == list:
process(obj)
# Good: Use isinstance
if isinstance(obj, list):
process(obj)
# Bad: Comparing to None with ==
if value == None:
process()
# Good: Use is
if value is None:
process()
# Bad: from module import *
from os.path import *
# Good: Explicit imports
from os.path import join, exists
# Bad: Bare except
try:
risky_operation()
except:
pass
# Good: Specific exception
try:
risky_operation()
except SpecificError as e:
logger.error(f"Operation failed: {e}")
Remember: Python code should be readable, explicit, and follow the principle of least surprise. When in doubt, prioritize clarity over cleverness.
Python Performance Optimization
Comprehensive guide to profiling, analyzing, and optimizing Python code for better performance, including CPU profiling, memory optimization, and implementation best practices.
When to Use This Skill
- Identifying performance bottlenecks in Python applications
- Reducing application latency and response times
- Optimizing CPU-intensive operations
- Reducing memory consumption and memory leaks
- Improving database query performance
- Optimizing I/O operations
- Speeding up data processing pipelines
- Implementing high-performance algorithms
- Profiling production applications
Core Concepts
1. Profiling Types
- CPU Profiling: Identify time-consuming functions
- Memory Profiling: Track memory allocation and leaks
- Line Profiling: Profile at line-by-line granularity
- Call Graph: Visualize function call relationships
2. Performance Metrics
- Execution Time: How long operations take
- Memory Usage: Peak and average memory consumption
- CPU Utilization: Processor usage patterns
- I/O Wait: Time spent on I/O operations
3. Optimization Strategies
- Algorithmic: Better algorithms and data structures
- Implementation: More efficient code patterns
- Parallelization: Multi-threading/processing
- Caching: Avoid redundant computation
- Native Extensions: C/Rust for critical paths
Quick Start
Basic Timing
import time
def measure_time():
"""Simple timing measurement."""
start = time.time()
# Your code here
result = sum(range(1000000))
elapsed = time.time() - start
print(f"Execution time: {elapsed:.4f} seconds")
return result
# Better: use timeit for accurate measurements
import timeit
execution_time = timeit.timeit(
"sum(range(1000000))",
number=100
)
print(f"Average time: {execution_time/100:.6f} seconds")
Detailed patterns and worked examples
Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.
Best Practices
- Profile before optimizing - Measure to find real bottlenecks
- Focus on hot paths - Optimize code that runs most frequently
- Use appropriate data structures - Dict for lookups, set for membership
- Avoid premature optimization - Clarity first, then optimize
- Use built-in functions - They're implemented in C
- Cache expensive computations - Use lru_cache
- Batch I/O operations - Reduce system calls
- Use generators for large datasets
- Consider NumPy for numerical operations
- Profile production code - Use py-spy for live systems
Common Pitfalls
- Optimizing without profiling
- Using global variables unnecessarily
- Not using appropriate data structures
- Creating unnecessary copies of data
- Not using connection pooling for databases
- Ignoring algorithmic complexity
- Over-optimizing rare code paths
- Not considering memory usage
Python Project Structure & Module Architecture
Design well-organized Python projects with clear module boundaries, explicit public interfaces, and maintainable directory structures. Good organization makes code discoverable and changes predictable.
When to Use This Skill
- Starting a new Python project from scratch
- Reorganizing an existing codebase for clarity
- Defining module public APIs with
__all__ - Deciding between flat and nested directory structures
- Determining test file placement strategies
- Creating reusable library packages
Core Concepts
1. Module Cohesion
Group related code that changes together. A module should have a single, clear purpose.
2. Explicit Interfaces
Define what's public with __all__. Everything not listed is an internal implementation detail.
3. Flat Hierarchies
Prefer shallow directory structures. Add depth only for genuine sub-domains.
4. Consistent Conventions
Apply naming and organization patterns uniformly across the project.
Quick Start
myproject/
├── src/
│ └── myproject/
│ ├── __init__.py
│ ├── services/
│ ├── models/
│ └── api/
├── tests/
├── pyproject.toml
└── README.md
Fundamental Patterns
Pattern 1: One Concept Per File
Each file should focus on a single concept or closely related set of functions. Consider splitting when a file:
- Handles multiple unrelated responsibilities
- Grows beyond 300-500 lines (varies by complexity)
- Contains classes that change for different reasons
# Good: Focused files
# user_service.py - User business logic
# user_repository.py - User data access
# user_models.py - User data structures
# Avoid: Kitchen sink files
# user.py - Contains service, repository, models, utilities...
Pattern 2: Explicit Public APIs with __all__
Define the public interface for every module. Unlisted members are internal implementation details.
# mypackage/services/__init__.py
from .user_service import UserService
from .order_service import OrderService
from .exceptions import ServiceError, ValidationError
__all__ = [
"UserService",
"OrderService",
"ServiceError",
"ValidationError",
]
# Internal helpers remain private by omission
# from .internal_helpers import _validate_input # Not exported
Pattern 3: Flat Directory Structure
Prefer minimal nesting. Deep hierarchies make imports verbose and navigation difficult.
# Preferred: Flat structure
project/
├── api/
│ ├── routes.py
│ └── middleware.py
├── services/
│ ├── user_service.py
│ └── order_service.py
├── models/
│ ├── user.py
│ └── order.py
└── utils/
└── validation.py
# Avoid: Deep nesting
project/core/internal/services/impl/user/
Add sub-packages only when there's a genuine sub-domain requiring isolation.
Pattern 4: Test File Organization
Choose one approach and apply it consistently throughout the project.
Option A: Colocated Tests
src/
├── user_service.py
├── test_user_service.py
├── order_service.py
└── test_order_service.py
Benefits: Tests live next to the code they verify. Easy to see coverage gaps.
Option B: Parallel Test Directory
src/
├── services/
│ ├── user_service.py
│ └── order_service.py
tests/
├── services/
│ ├── test_user_service.py
│ └── test_order_service.py
Benefits: Clean separation between production and test code. Standard for larger projects.
Advanced Patterns
Pattern 5: Package Initialization
Use __init__.py to provide a clean public interface for package consumers.
# mypackage/__init__.py
"""MyPackage - A library for doing useful things."""
from .core import MainClass, HelperClass
from .exceptions import PackageError, ConfigError
from .config import Settings
__all__ = [
"MainClass",
"HelperClass",
"PackageError",
"ConfigError",
"Settings",
]
__version__ = "1.0.0"
Consumers can then import directly from the package:
from mypackage import MainClass, Settings
Pattern 6: Layered Architecture
Organize code by architectural layer for clear separation of concerns.
myapp/
├── api/ # HTTP handlers, request/response
│ ├── routes/
│ └── middleware/
├── services/ # Business logic
├── repositories/ # Data access
├── models/ # Domain entities
├── schemas/ # API schemas (Pydantic)
└── config/ # Configuration
Each layer should only depend on layers below it, never above.
Pattern 7: Domain-Driven Structure
For complex applications, organize by business domain rather than technical layer.
ecommerce/
├── users/
│ ├── models.py
│ ├── services.py
│ ├── repository.py
│ └── api.py
├── orders/
│ ├── models.py
│ ├── services.py
│ ├── repository.py
│ └── api.py
└── shared/
├── database.py
└── exceptions.py
File and Module Naming
Conventions
- Use
snake_casefor all file and module names:user_repository.py - Avoid abbreviations that obscure meaning:
user_repository.pynotusr_repo.py - Match class names to file names:
UserServiceinuser_service.py
Import Style
Use absolute imports for clarity and reliability:
# Preferred: Absolute imports
from myproject.services import UserService
from myproject.models import User
# Avoid: Relative imports
from ..services import UserService
from . import models
Relative imports can break when modules are moved or reorganized.
Best Practices Summary
- Keep files focused - One concept per file, consider splitting at 300-500 lines (varies by complexity)
- Define
__all__explicitly - Make public interfaces clear - Prefer flat structures - Add depth only for genuine sub-domains
- Use absolute imports - More reliable and clearer
- Be consistent - Apply patterns uniformly across the project
- Match names to content - File names should describe their purpose
- Separate concerns - Keep layers distinct and dependencies flowing one direction
- Document your structure - Include a README explaining the organization
Python Resilience Patterns
Build fault-tolerant Python applications that gracefully handle transient failures, network issues, and service outages. Resilience patterns keep systems running when dependencies are unreliable.
When to Use This Skill
- Adding retry logic to external service calls
- Implementing timeouts for network operations
- Building fault-tolerant microservices
- Handling rate limiting and backpressure
- Creating infrastructure decorators
- Designing circuit breakers
Core Concepts
1. Transient vs Permanent Failures
Retry transient errors (network timeouts, temporary service issues). Don't retry permanent errors (invalid credentials, bad requests).
2. Exponential Backoff
Increase wait time between retries to avoid overwhelming recovering services.
3. Jitter
Add randomness to backoff to prevent thundering herd when many clients retry simultaneously.
4. Bounded Retries
Cap both attempt count and total duration to prevent infinite retry loops.
Quick Start
from tenacity import retry, stop_after_attempt, wait_exponential_jitter
@retry(
stop=stop_after_attempt(3),
wait=wait_exponential_jitter(initial=1, max=10),
)
def call_external_service(request: dict) -> dict:
return httpx.post("https://api.example.com", json=request).json()
Fundamental Patterns
Pattern 1: Basic Retry with Tenacity
Use the tenacity library for production-grade retry logic. For simpler cases, consider built-in retry functionality or a lightweight custom implementation.
from tenacity import (
retry,
stop_after_attempt,
stop_after_delay,
wait_exponential_jitter,
retry_if_exception_type,
)
TRANSIENT_ERRORS = (ConnectionError, TimeoutError, OSError)
@retry(
retry=retry_if_exception_type(TRANSIENT_ERRORS),
stop=stop_after_attempt(5) | stop_after_delay(60),
wait=wait_exponential_jitter(initial=1, max=30),
)
def fetch_data(url: str) -> dict:
"""Fetch data with automatic retry on transient failures."""
response = httpx.get(url, timeout=30)
response.raise_for_status()
return response.json()
Pattern 2: Retry Only Appropriate Errors
Whitelist specific transient exceptions. Never retry:
ValueError,TypeError- These are bugs, not transient issuesAuthenticationError- Invalid credentials won't become valid- HTTP 4xx errors (except 429) - Client errors are permanent
from tenacity import retry, retry_if_exception_type
import httpx
# Define what's retryable
RETRYABLE_EXCEPTIONS = (
ConnectionError,
TimeoutError,
httpx.ConnectTimeout,
httpx.ReadTimeout,
)
@retry(
retry=retry_if_exception_type(RETRYABLE_EXCEPTIONS),
stop=stop_after_attempt(3),
wait=wait_exponential_jitter(initial=1, max=10),
)
def resilient_api_call(endpoint: str) -> dict:
"""Make API call with retry on network issues."""
return httpx.get(endpoint, timeout=10).json()
Pattern 3: HTTP Status Code Retries
Retry specific HTTP status codes that indicate transient issues.
from tenacity import retry, retry_if_result, stop_after_attempt
import httpx
RETRY_STATUS_CODES = {429, 502, 503, 504}
def should_retry_response(response: httpx.Response) -> bool:
"""Check if response indicates a retryable error."""
return response.status_code in RETRY_STATUS_CODES
@retry(
retry=retry_if_result(should_retry_response),
stop=stop_after_attempt(3),
wait=wait_exponential_jitter(initial=1, max=10),
)
def http_request(method: str, url: str, **kwargs) -> httpx.Response:
"""Make HTTP request with retry on transient status codes."""
return httpx.request(method, url, timeout=30, **kwargs)
Pattern 4: Combined Exception and Status Retry
Handle both network exceptions and HTTP status codes.
from tenacity import (
retry,
retry_if_exception_type,
retry_if_result,
stop_after_attempt,
wait_exponential_jitter,
before_sleep_log,
)
import logging
import httpx
logger = logging.getLogger(__name__)
TRANSIENT_EXCEPTIONS = (
ConnectionError,
TimeoutError,
httpx.ConnectError,
httpx.ReadTimeout,
)
RETRY_STATUS_CODES = {429, 500, 502, 503, 504}
def is_retryable_response(response: httpx.Response) -> bool:
return response.status_code in RETRY_STATUS_CODES
@retry(
retry=(
retry_if_exception_type(TRANSIENT_EXCEPTIONS) |
retry_if_result(is_retryable_response)
),
stop=stop_after_attempt(5),
wait=wait_exponential_jitter(initial=1, max=30),
before_sleep=before_sleep_log(logger, logging.WARNING),
)
def robust_http_call(
method: str,
url: str,
**kwargs,
) -> httpx.Response:
"""HTTP call with comprehensive retry handling."""
return httpx.request(method, url, timeout=30, **kwargs)
Detailed worked examples and patterns
Detailed sections (starting with ## Advanced Patterns) live in references/details.md. Read that file when the navigation summary above is insufficient.
Best Practices Summary
- Retry only transient errors - Don't retry bugs or authentication failures
- Use exponential backoff - Give services time to recover
- Add jitter - Prevent thundering herd from synchronized retries
- Cap total duration -
stop_after_attempt(5) | stop_after_delay(60) - Log every retry - Silent retries hide systemic problems
- Use decorators - Keep retry logic separate from business logic
- Inject dependencies - Make infrastructure testable
- Set timeouts everywhere - Every network call needs a timeout
- Fail gracefully - Return cached/default values for non-critical paths
- Monitor retry rates - High retry rates indicate underlying issues
Python Resource Management
Manage resources deterministically using context managers. Resources like database connections, file handles, and network sockets should be released reliably, even when exceptions occur.
When to Use This Skill
- Managing database connections and connection pools
- Working with file handles and I/O
- Implementing custom context managers
- Building streaming responses with state
- Handling nested resource cleanup
- Creating async context managers
Core Concepts
1. Context Managers
The with statement ensures resources are released automatically, even on exceptions.
2. Protocol Methods
__enter__/__exit__ for sync, __aenter__/__aexit__ for async resource management.
3. Unconditional Cleanup
__exit__ always runs, regardless of whether an exception occurred.
4. Exception Handling
Return True from __exit__ to suppress exceptions, False to propagate them.
Quick Start
from contextlib import contextmanager
@contextmanager
def managed_resource():
resource = acquire_resource()
try:
yield resource
finally:
resource.cleanup()
with managed_resource() as r:
r.do_work()
Fundamental Patterns
Pattern 1: Class-Based Context Manager
Implement the context manager protocol for complex resources.
class DatabaseConnection:
"""Database connection with automatic cleanup."""
def __init__(self, dsn: str) -> None:
self._dsn = dsn
self._conn: Connection | None = None
def connect(self) -> None:
"""Establish database connection."""
self._conn = psycopg.connect(self._dsn)
def close(self) -> None:
"""Close connection if open."""
if self._conn is not None:
self._conn.close()
self._conn = None
def __enter__(self) -> "DatabaseConnection":
"""Enter context: connect and return self."""
self.connect()
return self
def __exit__(
self,
exc_type: type[BaseException] | None,
exc_val: BaseException | None,
exc_tb: TracebackType | None,
) -> None:
"""Exit context: always close connection."""
self.close()
# Usage with context manager (preferred)
with DatabaseConnection(dsn) as db:
result = db.execute(query)
# Manual management when needed
db = DatabaseConnection(dsn)
db.connect()
try:
result = db.execute(query)
finally:
db.close()
Pattern 2: Async Context Manager
For async resources, implement the async protocol.
class AsyncDatabasePool:
"""Async database connection pool."""
def __init__(self, dsn: str, min_size: int = 1, max_size: int = 10) -> None:
self._dsn = dsn
self._min_size = min_size
self._max_size = max_size
self._pool: asyncpg.Pool | None = None
async def __aenter__(self) -> "AsyncDatabasePool":
"""Create connection pool."""
self._pool = await asyncpg.create_pool(
self._dsn,
min_size=self._min_size,
max_size=self._max_size,
)
return self
async def __aexit__(
self,
exc_type: type[BaseException] | None,
exc_val: BaseException | None,
exc_tb: TracebackType | None,
) -> None:
"""Close all connections in pool."""
if self._pool is not None:
await self._pool.close()
async def execute(self, query: str, *args) -> list[dict]:
"""Execute query using pooled connection."""
async with self._pool.acquire() as conn:
return await conn.fetch(query, *args)
# Usage
async with AsyncDatabasePool(dsn) as pool:
users = await pool.execute("SELECT * FROM users WHERE active = $1", True)
Pattern 3: Using @contextmanager Decorator
Simplify context managers with the decorator for straightforward cases.
from contextlib import contextmanager, asynccontextmanager
import time
import structlog
logger = structlog.get_logger()
@contextmanager
def timed_block(name: str):
"""Time a block of code."""
start = time.perf_counter()
try:
yield
finally:
elapsed = time.perf_counter() - start
logger.info(f"{name} completed", duration_seconds=round(elapsed, 3))
# Usage
with timed_block("data_processing"):
process_large_dataset()
@asynccontextmanager
async def database_transaction(conn: AsyncConnection):
"""Manage database transaction."""
await conn.execute("BEGIN")
try:
yield conn
await conn.execute("COMMIT")
except BaseException: # also roll back on asyncio.CancelledError
await conn.execute("ROLLBACK")
raise
# Usage
async with database_transaction(conn) as tx:
await tx.execute("INSERT INTO users ...")
await tx.execute("INSERT INTO audit_log ...")
Pattern 4: Unconditional Resource Release
Always clean up resources in __exit__, regardless of exceptions.
class FileProcessor:
"""Process file with guaranteed cleanup."""
def __init__(self, path: str) -> None:
self._path = path
self._file: IO | None = None
self._temp_files: list[Path] = []
def __enter__(self) -> "FileProcessor":
self._file = open(self._path, "r")
return self
def __exit__(
self,
exc_type: type[BaseException] | None,
exc_val: BaseException | None,
exc_tb: TracebackType | None,
) -> None:
"""Clean up all resources unconditionally."""
# Close main file
if self._file is not None:
self._file.close()
# Clean up any temporary files
for temp_file in self._temp_files:
try:
temp_file.unlink()
except OSError:
pass # Best effort cleanup
# Return None/False to propagate any exception
Detailed worked examples and patterns
Detailed sections (starting with ## Advanced Patterns) live in references/details.md. Read that file when the navigation summary above is insufficient.
Best Practices Summary
- Always use context managers - For any resource that needs cleanup
- Clean up unconditionally -
__exit__runs even on exception - Don't suppress unexpectedly - Return
Falseunless suppression is intentional - Use @contextmanager - For simple resource patterns
- Implement both protocols - Support
withand manual management - Use ExitStack - For dynamic numbers of resources
- Accumulate efficiently - List + join, not string concatenation
- Track metrics - Time-to-first-byte matters for streaming
- Document behavior - Especially exception suppression
- Test cleanup paths - Verify resources are released on errors
Python Testing Patterns
Comprehensive guide to implementing robust testing strategies in Python using pytest, fixtures, mocking, parameterization, and test-driven development practices.
When to Use This Skill
- Writing unit tests for Python code
- Setting up test suites and test infrastructure
- Implementing test-driven development (TDD)
- Creating integration tests for APIs and services
- Mocking external dependencies and services
- Testing async code and concurrent operations
- Setting up continuous testing in CI/CD
- Implementing property-based testing
- Testing database operations
- Debugging failing tests
Core Concepts
1. Test Types
- Unit Tests: Test individual functions/classes in isolation
- Integration Tests: Test interaction between components
- Functional Tests: Test complete features end-to-end
- Performance Tests: Measure speed and resource usage
2. Test Structure (AAA Pattern)
- Arrange: Set up test data and preconditions
- Act: Execute the code under test
- Assert: Verify the results
3. Test Coverage
- Measure what code is exercised by tests
- Identify untested code paths
- Aim for meaningful coverage, not just high percentages
4. Test Isolation
- Tests should be independent
- No shared state between tests
- Each test should clean up after itself
Quick Start
# test_example.py
def add(a, b):
return a + b
def test_add():
"""Basic test example."""
result = add(2, 3)
assert result == 5
def test_add_negative():
"""Test with negative numbers."""
assert add(-1, 1) == 0
# Run with: pytest test_example.py
Detailed patterns and worked examples
Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.
Testing Best Practices
Test Organization
# tests/
# __init__.py
# conftest.py # Shared fixtures
# test_unit/ # Unit tests
# test_models.py
# test_utils.py
# test_integration/ # Integration tests
# test_api.py
# test_database.py
# test_e2e/ # End-to-end tests
# test_workflows.py
Test Naming Convention
A common pattern: test_<unit>_<scenario>_<expected_outcome>. Adapt to your team's preferences.
# Pattern: test_<unit>_<scenario>_<expected>
def test_create_user_with_valid_data_returns_user():
...
def test_create_user_with_duplicate_email_raises_conflict():
...
def test_get_user_with_unknown_id_returns_none():
...
# Good test names - clear and descriptive
def test_user_creation_with_valid_data():
"""Clear name describes what is being tested."""
pass
def test_login_fails_with_invalid_password():
"""Name describes expected behavior."""
pass
def test_api_returns_404_for_missing_resource():
"""Specific about inputs and expected outcomes."""
pass
# Bad test names - avoid these
def test_1(): # Not descriptive
pass
def test_user(): # Too vague
pass
def test_function(): # Doesn't explain what's tested
pass
Testing Retry Behavior
Verify that retry logic works correctly using mock side effects.
from unittest.mock import Mock
def test_retries_on_transient_error():
"""Test that service retries on transient failures."""
client = Mock()
# Fail twice, then succeed
client.request.side_effect = [
ConnectionError("Failed"),
ConnectionError("Failed"),
{"status": "ok"},
]
service = ServiceWithRetry(client, max_retries=3)
result = service.fetch()
assert result == {"status": "ok"}
assert client.request.call_count == 3
def test_gives_up_after_max_retries():
"""Test that service stops retrying after max attempts."""
client = Mock()
client.request.side_effect = ConnectionError("Failed")
service = ServiceWithRetry(client, max_retries=3)
with pytest.raises(ConnectionError):
service.fetch()
assert client.request.call_count == 3
def test_does_not_retry_on_permanent_error():
"""Test that permanent errors are not retried."""
client = Mock()
client.request.side_effect = ValueError("Invalid input")
service = ServiceWithRetry(client, max_retries=3)
with pytest.raises(ValueError):
service.fetch()
# Only called once - no retry for ValueError
assert client.request.call_count == 1
Mocking Time with Freezegun
Use freezegun to control time in tests for predictable time-dependent behavior.
from freezegun import freeze_time
from datetime import datetime, timedelta
@freeze_time("2026-01-15 10:00:00")
def test_token_expiry():
"""Test token expires at correct time."""
token = create_token(expires_in_seconds=3600)
assert token.expires_at == datetime(2026, 1, 15, 11, 0, 0)
@freeze_time("2026-01-15 10:00:00")
def test_is_expired_returns_false_before_expiry():
"""Test token is not expired when within validity period."""
token = create_token(expires_in_seconds=3600)
assert not token.is_expired()
@freeze_time("2026-01-15 12:00:00")
def test_is_expired_returns_true_after_expiry():
"""Test token is expired after validity period."""
token = Token(expires_at=datetime(2026, 1, 15, 11, 30, 0))
assert token.is_expired()
def test_with_time_travel():
"""Test behavior across time using freeze_time context."""
with freeze_time("2026-01-01") as frozen_time:
item = create_item()
assert item.created_at == datetime(2026, 1, 1)
# Move forward in time
frozen_time.move_to("2026-01-15")
assert item.age_days == 14
Test Markers
# test_markers.py
import pytest
@pytest.mark.slow
def test_slow_operation():
"""Mark slow tests."""
import time
time.sleep(2)
@pytest.mark.integration
def test_database_integration():
"""Mark integration tests."""
pass
@pytest.mark.skip(reason="Feature not implemented yet")
def test_future_feature():
"""Skip tests temporarily."""
pass
@pytest.mark.skipif(os.name == "nt", reason="Unix only test")
def test_unix_specific():
"""Conditional skip."""
pass
@pytest.mark.xfail(reason="Known bug #123")
def test_known_bug():
"""Mark expected failures."""
assert False
# Run with:
# pytest -m slow # Run only slow tests
# pytest -m "not slow" # Skip slow tests
# pytest -m integration # Run integration tests
Coverage Reporting
# Install coverage
pip install pytest-cov
# Run tests with coverage
pytest --cov=myapp tests/
# Generate HTML report
pytest --cov=myapp --cov-report=html tests/
# Fail if coverage below threshold
pytest --cov=myapp --cov-fail-under=80 tests/
# Show missing lines
pytest --cov=myapp --cov-report=term-missing tests/
For advanced patterns (async testing, monkeypatching, property-based testing, database testing, CI/CD integration, and configuration), see references/advanced-patterns.md
Python Testing Patterns
Comprehensive testing strategies for Python applications using pytest, TDD methodology, and best practices.
When to Activate
- Writing new Python code (follow TDD: red, green, refactor)
- Designing test suites for Python projects
- Reviewing Python test coverage
- Setting up testing infrastructure
Core Testing Philosophy
Test-Driven Development (TDD)
Always follow the TDD cycle:
- RED: Write a failing test for the desired behavior
- GREEN: Write minimal code to make the test pass
- REFACTOR: Improve code while keeping tests green
# Step 1: Write failing test (RED)
def test_add_numbers():
result = add(2, 3)
assert result == 5
# Step 2: Write minimal implementation (GREEN)
def add(a, b):
return a + b
# Step 3: Refactor if needed (REFACTOR)
Coverage Requirements
- Target: 80%+ code coverage
- Critical paths: 100% coverage required
- Use
pytest --covto measure coverage
pytest --cov=mypackage --cov-report=term-missing --cov-report=html
pytest Fundamentals
Basic Test Structure
import pytest
def test_addition():
"""Test basic addition."""
assert 2 + 2 == 4
def test_string_uppercase():
"""Test string uppercasing."""
text = "hello"
assert text.upper() == "HELLO"
def test_list_append():
"""Test list append."""
items = [1, 2, 3]
items.append(4)
assert 4 in items
assert len(items) == 4
Assertions
# Equality
assert result == expected
# Inequality
assert result != unexpected
# Truthiness
assert result # Truthy
assert not result # Falsy
assert result is True # Exactly True
assert result is False # Exactly False
assert result is None # Exactly None
# Membership
assert item in collection
assert item not in collection
# Comparisons
assert result > 0
assert 0 <= result <= 100
# Type checking
assert isinstance(result, str)
# Exception testing (preferred approach)
with pytest.raises(ValueError):
raise ValueError("error message")
# Check exception message
with pytest.raises(ValueError, match="invalid input"):
raise ValueError("invalid input provided")
# Check exception attributes
with pytest.raises(ValueError) as exc_info:
raise ValueError("error message")
assert str(exc_info.value) == "error message"
Fixtures
Basic Fixture Usage
import pytest
@pytest.fixture
def sample_data():
"""Fixture providing sample data."""
return {"name": "Alice", "age": 30}
def test_sample_data(sample_data):
"""Test using the fixture."""
assert sample_data["name"] == "Alice"
assert sample_data["age"] == 30
Fixture with Setup/Teardown
@pytest.fixture
def database():
"""Fixture with setup and teardown."""
# Setup
db = Database(":memory:")
db.create_tables()
db.insert_test_data()
yield db # Provide to test
# Teardown
db.close()
def test_database_query(database):
"""Test database operations."""
result = database.query("SELECT * FROM users")
assert len(result) > 0
Fixture Scopes
# Function scope (default) - runs for each test
@pytest.fixture
def temp_file():
with open("temp.txt", "w") as f:
yield f
os.remove("temp.txt")
# Module scope - runs once per module
@pytest.fixture(scope="module")
def module_db():
db = Database(":memory:")
db.create_tables()
yield db
db.close()
# Session scope - runs once per test session
@pytest.fixture(scope="session")
def shared_resource():
resource = ExpensiveResource()
yield resource
resource.cleanup()
Fixture with Parameters
@pytest.fixture(params=[1, 2, 3])
def number(request):
"""Parameterized fixture."""
return request.param
def test_numbers(number):
"""Test runs 3 times, once for each parameter."""
assert number > 0
Using Multiple Fixtures
@pytest.fixture
def user():
return User(id=1, name="Alice")
@pytest.fixture
def admin():
return User(id=2, name="Admin", role="admin")
def test_user_admin_interaction(user, admin):
"""Test using multiple fixtures."""
assert admin.can_manage(user)
Autouse Fixtures
@pytest.fixture(autouse=True)
def reset_config():
"""Automatically runs before every test."""
Config.reset()
yield
Config.cleanup()
def test_without_fixture_call():
# reset_config runs automatically
assert Config.get_setting("debug") is False
Conftest.py for Shared Fixtures (Flask)
# tests/conftest.py
import pytest
@pytest.fixture
def client():
"""Shared fixture for all tests."""
app = create_app(testing=True)
with app.test_client() as client:
yield client
@pytest.fixture
def auth_headers(client):
"""Generate auth headers for API testing."""
response = client.post("/api/login", json={
"username": "test",
"password": "test"
})
token = response.json["token"]
return {"Authorization": f"Bearer {token}"}
Parametrization
Basic Parametrization
@pytest.mark.parametrize("input,expected", [
("hello", "HELLO"),
("world", "WORLD"),
("PyThOn", "PYTHON"),
])
def test_uppercase(input, expected):
"""Test runs 3 times with different inputs."""
assert input.upper() == expected
Multiple Parameters
@pytest.mark.parametrize("a,b,expected", [
(2, 3, 5),
(0, 0, 0),
(-1, 1, 0),
(100, 200, 300),
])
def test_add(a, b, expected):
"""Test addition with multiple inputs."""
assert add(a, b) == expected
Parametrize with IDs
@pytest.mark.parametrize("input,expected", [
("valid@email.com", True),
("invalid", False),
("@no-domain.com", False),
], ids=["valid-email", "missing-at", "missing-domain"])
def test_email_validation(input, expected):
"""Test email validation with readable test IDs."""
assert is_valid_email(input) is expected
Parametrized Fixtures
@pytest.fixture(params=["sqlite", "postgresql", "mysql"])
def db(request):
"""Test against multiple database backends."""
if request.param == "sqlite":
return Database(":memory:")
elif request.param == "postgresql":
return Database("postgresql://localhost/test")
elif request.param == "mysql":
return Database("mysql://localhost/test")
def test_database_operations(db):
"""Test runs 3 times, once for each database."""
result = db.query("SELECT 1")
assert result is not None
Markers and Test Selection
Custom Markers
# Mark slow tests
@pytest.mark.slow
def test_slow_operation():
time.sleep(5)
# Mark integration tests
@pytest.mark.integration
def test_api_integration():
response = requests.get("https://api.example.com")
assert response.status_code == 200
# Mark unit tests
@pytest.mark.unit
def test_unit_logic():
assert calculate(2, 3) == 5
Run Specific Tests
# Run only fast tests
pytest -m "not slow"
# Run only integration tests
pytest -m integration
# Run integration or slow tests
pytest -m "integration or slow"
# Run tests marked as unit but not slow
pytest -m "unit and not slow"
Configure Markers in pytest.ini
[pytest]
markers =
slow: marks tests as slow
integration: marks tests as integration tests
unit: marks tests as unit tests
django: marks tests as requiring Django
Mocking and Patching
Mocking Functions
from unittest.mock import patch, Mock
@patch("mypackage.external_api_call")
def test_with_mock(api_call_mock):
"""Test with mocked external API."""
api_call_mock.return_value = {"status": "success"}
result = my_function()
api_call_mock.assert_called_once()
assert result["status"] == "success"
Mocking Return Values
@patch("mypackage.Database.connect")
def test_database_connection(connect_mock):
"""Test with mocked database connection."""
connect_mock.return_value = MockConnection()
db = Database()
db.connect()
connect_mock.assert_called_once_with("localhost")
Mocking Exceptions
@patch("mypackage.api_call")
def test_api_error_handling(api_call_mock):
"""Test error handling with mocked exception."""
api_call_mock.side_effect = ConnectionError("Network error")
with pytest.raises(ConnectionError):
api_call()
api_call_mock.assert_called_once()
Mocking Context Managers
@patch("builtins.open", new_callable=mock_open)
def test_file_reading(mock_file):
"""Test file reading with mocked open."""
mock_file.return_value.read.return_value = "file content"
result = read_file("test.txt")
mock_file.assert_called_once_with("test.txt", "r")
assert result == "file content"
Using Autospec
@patch("mypackage.DBConnection", autospec=True)
def test_autospec(db_mock):
"""Test with autospec to catch API misuse."""
db = db_mock.return_value
db.query("SELECT * FROM users")
# This would fail if DBConnection doesn't have query method
db_mock.assert_called_once()
Mock Class Instances
class TestUserService:
@patch("mypackage.UserRepository")
def test_create_user(self, repo_mock):
"""Test user creation with mocked repository."""
repo_mock.return_value.save.return_value = User(id=1, name="Alice")
service = UserService(repo_mock.return_value)
user = service.create_user(name="Alice")
assert user.name == "Alice"
repo_mock.return_value.save.assert_called_once()
Mock Property
@pytest.fixture
def mock_config():
"""Create a mock with a property."""
config = Mock()
type(config).debug = PropertyMock(return_value=True)
type(config).api_key = PropertyMock(return_value="test-key")
return config
def test_with_mock_config(mock_config):
"""Test with mocked config properties."""
assert mock_config.debug is True
assert mock_config.api_key == "test-key"
Testing Async Code
Async Tests with pytest-asyncio
import pytest
@pytest.mark.asyncio
async def test_async_function():
"""Test async function."""
result = await async_add(2, 3)
assert result == 5
@pytest.mark.asyncio
async def test_async_with_fixture(async_client):
"""Test async with async fixture."""
response = await async_client.get("/api/users")
assert response.status_code == 200
Async Fixture (FastAPI)
ASGITransport does not run lifespan events. Install asgi-lifespan and wrap the client in LifespanManager when the app uses startup or shutdown handlers. See FastAPI's async testing guidance.
import pytest
import pytest_asyncio
from asgi_lifespan import LifespanManager
from httpx import ASGITransport, AsyncClient
@pytest_asyncio.fixture
async def async_client():
"""Async fixture providing async test client with lifespan events."""
app = create_app() # Returns a FastAPI application
async with LifespanManager(app) as manager:
async with AsyncClient(transport=ASGITransport(app=manager.app), base_url="http://test") as client:
yield client
@pytest.mark.asyncio
async def test_api_endpoint(async_client):
"""Test using async fixture."""
response = await async_client.get("/api/data")
assert response.status_code == 200
Mocking Async Functions
@pytest.mark.asyncio
@patch("mypackage.async_api_call")
async def test_async_mock(api_call_mock):
"""Test async function with mock."""
api_call_mock.return_value = {"status": "ok"}
result = await my_async_function()
api_call_mock.assert_awaited_once()
assert result["status"] == "ok"
Testing Exceptions
Testing Expected Exceptions
def test_divide_by_zero():
"""Test that dividing by zero raises ZeroDivisionError."""
with pytest.raises(ZeroDivisionError):
divide(10, 0)
def test_custom_exception():
"""Test custom exception with message."""
with pytest.raises(ValueError, match="invalid input"):
validate_input("invalid")
Testing Exception Attributes
def test_exception_with_details():
"""Test exception with custom attributes."""
with pytest.raises(CustomError) as exc_info:
raise CustomError("error", code=400)
assert exc_info.value.code == 400
assert "error" in str(exc_info.value)
Testing Side Effects
Testing File Operations
import tempfile
import os
def test_file_processing():
"""Test file processing with temp file."""
with tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.txt') as f:
f.write("test content")
temp_path = f.name
try:
result = process_file(temp_path)
assert result == "processed: test content"
finally:
os.unlink(temp_path)
Testing with pytest's tmp_path Fixture
def test_with_tmp_path(tmp_path):
"""Test using pytest's built-in temp path fixture."""
test_file = tmp_path / "test.txt"
test_file.write_text("hello world")
result = process_file(str(test_file))
assert result == "hello world"
# tmp_path automatically cleaned up
Testing with tmpdir Fixture
def test_with_tmpdir(tmpdir):
"""Test using pytest's tmpdir fixture."""
test_file = tmpdir.join("test.txt")
test_file.write("data")
result = process_file(str(test_file))
assert result == "data"
Test Organization
Directory Structure
tests/
├── conftest.py # Shared fixtures
├── __init__.py
├── unit/ # Unit tests
│ ├── __init__.py
│ ├── test_models.py
│ ├── test_utils.py
│ └── test_services.py
├── integration/ # Integration tests
│ ├── __init__.py
│ ├── test_api.py
│ └── test_database.py
└── e2e/ # End-to-end tests
├── __init__.py
└── test_user_flow.py
Test Classes
class TestUserService:
"""Group related tests in a class."""
@pytest.fixture(autouse=True)
def setup(self):
"""Setup runs before each test in this class."""
self.service = UserService()
def test_create_user(self):
"""Test user creation."""
user = self.service.create_user("Alice")
assert user.name == "Alice"
def test_delete_user(self):
"""Test user deletion."""
user = User(id=1, name="Bob")
self.service.delete_user(user)
assert not self.service.user_exists(1)
Best Practices
DO
- Follow TDD: Write tests before code (red-green-refactor)
- Test one thing: Each test should verify a single behavior
- Use descriptive names:
test_user_login_with_invalid_credentials_fails - Use fixtures: Eliminate duplication with fixtures
- Mock external dependencies: Don't depend on external services
- Test edge cases: Empty inputs, None values, boundary conditions
- Aim for 80%+ coverage: Focus on critical paths
- Keep tests fast: Use marks to separate slow tests
DON'T
- Don't test implementation: Test behavior, not internals
- Don't use complex conditionals in tests: Keep tests simple
- Don't ignore test failures: All tests must pass
- Don't test third-party code: Trust libraries to work
- Don't share state between tests: Tests should be independent
- Don't catch exceptions in tests: Use
pytest.raises - Don't use print statements: Use assertions and pytest output
- Don't write tests that are too brittle: Avoid over-specific mocks
Common Patterns
Testing API Endpoints (Flask)
@pytest.fixture
def client():
app = create_app(testing=True)
return app.test_client()
def test_get_user(client):
response = client.get("/api/users/1")
assert response.status_code == 200
assert response.json["id"] == 1
def test_create_user(client):
response = client.post("/api/users", json={
"name": "Alice",
"email": "alice@example.com"
})
assert response.status_code == 201
assert response.json["name"] == "Alice"
Testing Database Operations
@pytest.fixture
def db_session():
"""Create a test database session."""
session = Session(bind=engine)
session.begin_nested()
yield session
session.rollback()
session.close()
def test_create_user(db_session):
user = User(name="Alice", email="alice@example.com")
db_session.add(user)
db_session.commit()
retrieved = db_session.query(User).filter_by(name="Alice").first()
assert retrieved.email == "alice@example.com"
Testing Class Methods
class TestCalculator:
@pytest.fixture
def calculator(self):
return Calculator()
def test_add(self, calculator):
assert calculator.add(2, 3) == 5
def test_divide_by_zero(self, calculator):
with pytest.raises(ZeroDivisionError):
calculator.divide(10, 0)
pytest Configuration
pytest.ini
[pytest]
testpaths = tests
python_files = test_*.py
python_classes = Test*
python_functions = test_*
addopts =
--strict-markers
--disable-warnings
--cov=mypackage
--cov-report=term-missing
--cov-report=html
markers =
slow: marks tests as slow
integration: marks tests as integration tests
unit: marks tests as unit tests
pyproject.toml
[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
python_classes = ["Test*"]
python_functions = ["test_*"]
addopts = [
"--strict-markers",
"--cov=mypackage",
"--cov-report=term-missing",
"--cov-report=html",
]
markers = [
"slow: marks tests as slow",
"integration: marks tests as integration tests",
"unit: marks tests as unit tests",
]
Running Tests
# Run all tests
pytest
# Run specific file
pytest tests/test_utils.py
# Run specific test
pytest tests/test_utils.py::test_function
# Run with verbose output
pytest -v
# Run with coverage
pytest --cov=mypackage --cov-report=html
# Run only fast tests
pytest -m "not slow"
# Run until first failure
pytest -x
# Run and stop on N failures
pytest --maxfail=3
# Run last failed tests
pytest --lf
# Run tests with pattern
pytest -k "test_user"
# Run with debugger on failure
pytest --pdb
Quick Reference
| Pattern | Usage |
|---|---|
pytest.raises() | Test expected exceptions |
@pytest.fixture() | Create reusable test fixtures |
@pytest.mark.parametrize() | Run tests with multiple inputs |
@pytest.mark.slow | Mark slow tests |
pytest -m "not slow" | Skip slow tests |
@patch() | Mock functions and classes |
tmp_path fixture | Automatic temp directory |
pytest --cov | Generate coverage report |
assert | Simple and readable assertions |
Remember: Tests are code too. Keep them clean, readable, and maintainable. Good tests catch bugs; great tests prevent them.
Python Type Safety
Leverage Python's type system to catch errors at static analysis time. Type annotations serve as enforced documentation that tooling validates automatically.
When to Use This Skill
- Adding type hints to existing code
- Creating generic, reusable classes
- Defining structural interfaces with protocols
- Configuring mypy or pyright for strict checking
- Understanding type narrowing and guards
- Building type-safe APIs and libraries
Core Concepts
1. Type Annotations
Declare expected types for function parameters, return values, and variables.
2. Generics
Write reusable code that preserves type information across different types.
3. Protocols
Define structural interfaces without inheritance (duck typing with type safety).
4. Type Narrowing
Use guards and conditionals to narrow types within code blocks.
Quick Start
def get_user(user_id: str) -> User | None:
"""Return type makes 'might not exist' explicit."""
...
# Type checker enforces handling None case
user = get_user("123")
if user is None:
raise UserNotFoundError("123")
print(user.name) # Type checker knows user is User here
Fundamental Patterns
Pattern 1: Annotate All Public Signatures
Every public function, method, and class should have type annotations.
def get_user(user_id: str) -> User:
"""Retrieve user by ID."""
...
def process_batch(
items: list[Item],
max_workers: int = 4,
) -> BatchResult[ProcessedItem]:
"""Process items concurrently."""
...
class UserRepository:
def __init__(self, db: Database) -> None:
self._db = db
async def find_by_id(self, user_id: str) -> User | None:
"""Return User if found, None otherwise."""
...
async def find_by_email(self, email: str) -> User | None:
...
async def save(self, user: User) -> User:
"""Save and return user with generated ID."""
...
Use mypy --strict or pyright in CI to catch type errors early. For existing projects, enable strict mode incrementally using per-module overrides.
Pattern 2: Use Modern Union Syntax
Python 3.10+ provides cleaner union syntax.
# Preferred (3.10+)
def find_user(user_id: str) -> User | None:
...
def parse_value(v: str) -> int | float | str:
...
# Older style (still valid, needed for 3.9)
from typing import Optional, Union
def find_user(user_id: str) -> Optional[User]:
...
Pattern 3: Type Narrowing with Guards
Use conditionals to narrow types for the type checker.
def process_user(user_id: str) -> UserData:
user = find_user(user_id)
if user is None:
raise UserNotFoundError(f"User {user_id} not found")
# Type checker knows user is User here, not User | None
return UserData(
name=user.name,
email=user.email,
)
def process_items(items: list[Item | None]) -> list[ProcessedItem]:
# Filter and narrow types
valid_items = [item for item in items if item is not None]
# valid_items is now list[Item]
return [process(item) for item in valid_items]
Pattern 4: Generic Classes
Create type-safe reusable containers.
from typing import TypeVar, Generic
T = TypeVar("T")
E = TypeVar("E", bound=Exception)
class Result(Generic[T, E]):
"""Represents either a success value or an error."""
def __init__(
self,
value: T | None = None,
error: E | None = None,
) -> None:
if (value is None) == (error is None):
raise ValueError("Exactly one of value or error must be set")
self._value = value
self._error = error
@property
def is_success(self) -> bool:
return self._error is None
@property
def is_failure(self) -> bool:
return self._error is not None
def unwrap(self) -> T:
"""Get value or raise the error."""
if self._error is not None:
raise self._error
return self._value # type: ignore[return-value]
def unwrap_or(self, default: T) -> T:
"""Get value or return default."""
if self._error is not None:
return default
return self._value # type: ignore[return-value]
# Usage preserves types
def parse_config(path: str) -> Result[Config, ConfigError]:
try:
return Result(value=Config.from_file(path))
except ConfigError as e:
return Result(error=e)
result = parse_config("config.yaml")
if result.is_success:
config = result.unwrap() # Type: Config
Detailed worked examples and patterns
Detailed sections (starting with ## Advanced Patterns) live in references/details.md. Read that file when the navigation summary above is insufficient.
Best Practices Summary
- Annotate all public APIs - Functions, methods, class attributes
- Use
T | None- Modern union syntax overOptional[T] - Run strict type checking -
mypy --strictin CI - Use generics - Preserve type info in reusable code
- Define protocols - Structural typing for interfaces
- Narrow types - Use guards to help the type checker
- Bound type vars - Restrict generics to meaningful types
- Create type aliases - Meaningful names for complex types
- Minimize
Any- Use specific types or generics.Anyis acceptable for truly dynamic data or when interfacing with untyped third-party code - Document with types - Types are enforceable documentation
Temporal Python Testing Strategies
Comprehensive testing approaches for Temporal workflows using pytest, progressive disclosure resources for specific testing scenarios.
When to Use This Skill
- Unit testing workflows - Fast tests with time-skipping
- Integration testing - Workflows with mocked activities
- Replay testing - Validate determinism against production histories
- Local development - Set up Temporal server and pytest
- CI/CD integration - Automated testing pipelines
- Coverage strategies - Achieve ≥80% test coverage
Testing Philosophy
Recommended Approach (Source: docs.temporal.io/develop/python/testing-suite):
- Write majority as integration tests
- Use pytest with async fixtures
- Time-skipping enables fast feedback (month-long workflows → seconds)
- Mock activities to isolate workflow logic
- Validate determinism with replay testing
Three Test Types:
- Unit: Workflows with time-skipping, activities with ActivityEnvironment
- Integration: Workers with mocked activities
- End-to-end: Full Temporal server with real activities (use sparingly)
Available Resources
This skill provides detailed guidance through progressive disclosure. Load specific resources based on your testing needs:
Unit Testing Resources
File: resources/unit-testing.md
When to load: Testing individual workflows or activities in isolation
Contains:
- WorkflowEnvironment with time-skipping
- ActivityEnvironment for activity testing
- Fast execution of long-running workflows
- Manual time advancement patterns
- pytest fixtures and patterns
Integration Testing Resources
File: resources/integration-testing.md
When to load: Testing workflows with mocked external dependencies
Contains:
- Activity mocking strategies
- Error injection patterns
- Multi-activity workflow testing
- Signal and query testing
- Coverage strategies
Replay Testing Resources
File: resources/replay-testing.md
When to load: Validating determinism or deploying workflow changes
Contains:
- Determinism validation
- Production history replay
- CI/CD integration patterns
- Version compatibility testing
Local Development Resources
File: resources/local-setup.md
When to load: Setting up development environment
Contains:
- Docker Compose configuration
- pytest setup and configuration
- Coverage tool integration
- Development workflow
Quick Start Guide
Basic Workflow Test
import pytest
from temporalio.testing import WorkflowEnvironment
from temporalio.worker import Worker
@pytest.fixture
async def workflow_env():
env = await WorkflowEnvironment.start_time_skipping()
yield env
await env.shutdown()
@pytest.mark.asyncio
async def test_workflow(workflow_env):
async with Worker(
workflow_env.client,
task_queue="test-queue",
workflows=[YourWorkflow],
activities=[your_activity],
):
result = await workflow_env.client.execute_workflow(
YourWorkflow.run,
args,
id="test-wf-id",
task_queue="test-queue",
)
assert result == expected
Basic Activity Test
from temporalio.testing import ActivityEnvironment
async def test_activity():
env = ActivityEnvironment()
result = await env.run(your_activity, "test-input")
assert result == expected_output
Coverage Targets
Recommended Coverage (Source: docs.temporal.io best practices):
- Workflows: ≥80% logic coverage
- Activities: ≥80% logic coverage
- Integration: Critical paths with mocked activities
- Replay: All workflow versions before deployment
Key Testing Principles
- Time-Skipping - Month-long workflows test in seconds
- Mock Activities - Isolate workflow logic from external dependencies
- Replay Testing - Validate determinism before deployment
- High Coverage - ≥80% target for production workflows
- Fast Feedback - Unit tests run in milliseconds
How to Use Resources
Load specific resource when needed:
- "Show me unit testing patterns" → Load
resources/unit-testing.md - "How do I mock activities?" → Load
resources/integration-testing.md - "Setup local Temporal server" → Load
resources/local-setup.md - "Validate determinism" → Load
resources/replay-testing.md
Additional References
- Python SDK Testing: docs.temporal.io/develop/python/testing-suite
- Testing Patterns: github.com/temporalio/temporal/blob/main/docs/development/testing.md
- Python Samples: github.com/temporalio/samples-python
UV Package Manager
Comprehensive guide to using uv, an extremely fast Python package installer and resolver written in Rust, for modern Python project management and dependency workflows.
When to Use This Skill
- Setting up new Python projects quickly
- Managing Python dependencies faster than pip
- Creating and managing virtual environments
- Installing Python interpreters
- Resolving dependency conflicts efficiently
- Migrating from pip/pip-tools/poetry
- Speeding up CI/CD pipelines
- Managing monorepo Python projects
- Working with lockfiles for reproducible builds
- Optimizing Docker builds with Python dependencies
Core Concepts
1. What is uv?
- Ultra-fast package installer: 10-100x faster than pip
- Written in Rust: Leverages Rust's performance
- Drop-in pip replacement: Compatible with pip workflows
- Virtual environment manager: Create and manage venvs
- Python installer: Download and manage Python versions
- Resolver: Advanced dependency resolution
- Lockfile support: Reproducible installations
2. Key Features
- Blazing fast installation speeds
- Disk space efficient with global cache
- Compatible with pip, pip-tools, poetry
- Comprehensive dependency resolution
- Cross-platform support (Linux, macOS, Windows)
- No Python required for installation
- Built-in virtual environment support
3. UV vs Traditional Tools
- vs pip: 10-100x faster, better resolver
- vs pip-tools: Faster, simpler, better UX
- vs poetry: Faster, less opinionated, lighter
- vs conda: Faster, Python-focused
Installation
Quick Install
# Using Homebrew (macOS or Linux)
brew install uv
# Using Cargo (if you have Rust)
cargo install --locked uv
On Windows, use WinGet:
winget install --id=astral-sh.uv -e
Use the official installation guide for other supported installation methods.
Verify Installation
uv --version
# uv 0.x.x
Quick Start
Create a New Project
# Create new project with virtual environment
uv init my-project
cd my-project
# Or create in current directory
uv init .
# Initialize creates:
# - .python-version (Python version)
# - pyproject.toml (project config)
# - README.md
# - .gitignore
Install Dependencies
# Install packages (creates venv if needed)
uv add requests pandas
# Install dev dependencies
uv add --dev pytest black ruff
# Install from requirements.txt
uv pip install -r requirements.txt
# Install from pyproject.toml
uv sync
Virtual Environment Management
Pattern 1: Creating Virtual Environments
# Create virtual environment with uv
uv venv
# Create with specific Python version
uv venv --python 3.12
# Create with custom name
uv venv my-env
# Create with system site packages
uv venv --system-site-packages
# Specify location
uv venv /path/to/venv
Pattern 2: Activating Virtual Environments
# Linux/macOS
source .venv/bin/activate
# Windows (Command Prompt)
.venv\Scripts\activate.bat
# Windows (PowerShell)
.venv\Scripts\Activate.ps1
# Or use uv run (no activation needed)
uv run python script.py
uv run pytest
Pattern 3: Using uv run
# Run Python script (auto-activates venv)
uv run python app.py
# Run installed CLI tool
uv run black .
uv run pytest
# Run with specific Python version
uv run --python 3.11 python script.py
# Pass arguments
uv run python script.py --arg value
Package Management
Pattern 4: Adding Dependencies
# Add package (adds to pyproject.toml)
uv add requests
# Add with version constraint
uv add "django>=4.0,<5.0"
# Add multiple packages
uv add numpy pandas matplotlib
# Add dev dependency
uv add --dev pytest pytest-cov
# Add optional dependency group
uv add --optional docs sphinx
# Add from git
uv add git+https://github.com/user/repo.git
# Add from git with specific ref
uv add git+https://github.com/user/repo.git@v1.0.0
# Add from local path
uv add ./local-package
# Add editable local package
uv add -e ./local-package
Pattern 5: Removing Dependencies
# Remove package
uv remove requests
# Remove dev dependency
uv remove --dev pytest
# Remove multiple packages
uv remove numpy pandas matplotlib
Pattern 6: Upgrading Dependencies
# Upgrade specific package
uv add --upgrade requests
# Upgrade all packages
uv sync --upgrade
# Upgrade package to latest
uv add --upgrade requests
# Show what would be upgraded
uv tree --outdated
Pattern 7: Locking Dependencies
# Generate uv.lock file
uv lock
# Update lock file
uv lock --upgrade
# Lock without installing
uv lock --no-install
# Lock specific package
uv lock --upgrade-package requests
Python Version Management
Pattern 8: Installing Python Versions
# Install Python version
uv python install 3.12
# Install multiple versions
uv python install 3.11 3.12 3.13
# Install latest version
uv python install
# List installed versions
uv python list
# Find available versions
uv python list --all-versions
Pattern 9: Setting Python Version
# Set Python version for project
uv python pin 3.12
# This creates/updates .python-version file
# Use specific Python version for command
uv --python 3.11 run python script.py
# Create venv with specific version
uv venv --python 3.12
Project Configuration
Pattern 10: pyproject.toml with uv
[project]
name = "my-project"
version = "0.1.0"
description = "My awesome project"
readme = "README.md"
requires-python = ">=3.8"
dependencies = [
"requests>=2.31.0",
"pydantic>=2.0.0",
"click>=8.1.0",
]
[project.optional-dependencies]
dev = [
"pytest>=7.4.0",
"pytest-cov>=4.1.0",
"black>=23.0.0",
"ruff>=0.1.0",
"mypy>=1.5.0",
]
docs = [
"sphinx>=7.0.0",
"sphinx-rtd-theme>=1.3.0",
]
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.uv]
dev-dependencies = [
# Additional dev dependencies managed by uv
]
[tool.uv.sources]
# Custom package sources
my-package = { git = "https://github.com/user/repo.git" }
Pattern 11: Using uv with Existing Projects
# Migrate from requirements.txt
uv add -r requirements.txt
# Migrate from poetry
# Already have pyproject.toml, just use:
uv sync
# Export to requirements.txt
uv pip freeze > requirements.txt
# Export with hashes
uv pip freeze --require-hashes > requirements.txt
For advanced workflows including Docker integration, lockfile management, performance optimization, tool comparison, common workflows, tool integration, troubleshooting, best practices, migration guides, and command reference, see references/advanced-patterns.md
Sponsors
Details
- Marketplace
- marslanalvi/SkillsStack
- Source
- plugins/engineering/python
- Category
- Engineering
- Keywords
- engineering, python
More from marslanalvi/SkillsStack
All 45- engineering-systems-languagesv1.0.0
- engineering-testingv1.0.0
- marketing-content-socialv1.0.0
- marketing-growth-adsv1.0.0
- marketing-seov1.0.0
- productivity-automation-personalv1.0.0
- productivity-documents-officev1.0.0
- productivity-thinking-planningv1.0.0
- productivity-writing-communicationv1.0.0
- security-appsec-reviewv1.0.0
- security-blockchain-web3v1.0.0
- security-compliance-privacyv1.0.0
- security-fuzzingv1.0.0
- security-offensive-securityv1.0.0
- ai-agent-engineeringv1.0.0
- ai-coding-assistant-workflowv1.0.0
- ai-llm-applicationsv1.0.0
- ai-machine-learningv1.0.0
- ai-skill-authoringv1.0.0
- business-legal-industryv1.0.0
- business-operations-financev1.0.0
- business-strategy-salesv1.0.0
- data-data-engineering-analyticsv1.0.0
- data-researchv1.0.0