
llm-engineering-skills
Collection of LLM engineering skills for PyTorch, Transformers, LoRA, and MLX
- GitHub stars
- 25
Install
/plugin install llm-engineering-skills@llm-engineering-skillsRun it inside Claude Code after adding the marketplace with /plugin marketplace add itsmostafa/llm-engineering-skills.
Contents
Discovered from sourceSkills 9
Building Agents
Agents are systems where LLMs dynamically direct their own processes and tool usage. This skill covers when to use agents vs workflows, common architectural patterns, and practical implementation guidance.
Table of Contents
- Agents vs Workflows
- Workflow Patterns
- Agent Architectures
- ReAct Pattern
- Tool Design
- External Context Protocols
- Best Practices
- References
Agents vs Workflows
| Aspect | Workflows | Agents |
|---|---|---|
| Control flow | Predefined code paths | LLM determines next step |
| Predictability | High - deterministic steps | Lower - dynamic decisions |
| Complexity | Simpler to debug and test | More complex, harder to predict |
| Best for | Well-defined, repeatable tasks | Open-ended, adaptive problems |
Key principle: Start with the simplest solution. Use workflows when the task is predictable; use agents when flexibility is required.
Workflow Patterns
1. Prompt Chaining
Decompose tasks into sequential LLM calls, where each step's output feeds the next.
async def prompt_chain(input_text):
# Step 1: Extract key information
extracted = await llm.generate(
"Extract the main entities and relationships from: " + input_text
)
# Step 2: Analyze
analysis = await llm.generate(
"Analyze these entities for patterns: " + extracted
)
# Step 3: Generate output
return await llm.generate(
"Based on this analysis, provide recommendations: " + analysis
)
Use when: Tasks naturally decompose into fixed sequential steps.
2. Routing
Classify inputs and direct them to specialized handlers.
async def route_request(user_input):
# Classify the input
category = await llm.generate(
f"Classify this request into one of: [billing, technical, general]\n{user_input}"
)
handlers = {
"billing": handle_billing,
"technical": handle_technical,
"general": handle_general,
}
return await handlers[category.strip()](user_input)
Use when: Different input types need fundamentally different processing.
3. Parallelization
Run multiple LLM calls concurrently for independent subtasks.
import asyncio
async def parallel_analysis(document):
# Run independent analyses in parallel
results = await asyncio.gather(
llm.generate(f"Summarize: {document}"),
llm.generate(f"Extract key facts: {document}"),
llm.generate(f"Identify sentiment: {document}"),
)
summary, facts, sentiment = results
return {"summary": summary, "facts": facts, "sentiment": sentiment}
Variants:
- Sectioning: Break task into parallel subtasks
- Voting: Run same prompt multiple times, aggregate results
4. Orchestrator-Workers
Central LLM decomposes tasks and delegates to worker LLMs.
class Orchestrator:
async def run(self, task):
# Break down the task
subtasks = await self.plan(task)
# Delegate to workers
results = []
for subtask in subtasks:
worker_result = await self.delegate(subtask)
results.append(worker_result)
# Synthesize results
return await self.synthesize(results)
async def plan(self, task):
response = await llm.generate(
f"Break this task into subtasks:\n{task}\n\nReturn as JSON array."
)
return json.loads(response)
async def delegate(self, subtask):
return await llm.generate(f"Complete this subtask:\n{subtask}")
async def synthesize(self, results):
return await llm.generate(
f"Combine these results into a coherent response:\n{results}"
)
Use when: Tasks require dynamic decomposition that can't be predetermined.
5. Evaluator-Optimizer
One LLM generates, another evaluates and requests improvements.
async def generate_with_feedback(task, max_iterations=3):
response = await llm.generate(f"Complete this task:\n{task}")
for _ in range(max_iterations):
evaluation = await llm.generate(
f"Evaluate this response for quality and correctness:\n{response}\n"
"If improvements needed, specify them. Otherwise respond 'APPROVED'."
)
if "APPROVED" in evaluation:
return response
response = await llm.generate(
f"Improve this response based on feedback:\n"
f"Original: {response}\nFeedback: {evaluation}"
)
return response
Use when: Output quality is critical and can be objectively evaluated.
Agent Architectures
Autonomous Agent Loop
Agents operate in a loop: observe, think, act, repeat.
class Agent:
def __init__(self, tools: list, system_prompt: str):
self.tools = {t.name: t for t in tools}
self.system_prompt = system_prompt
async def run(self, task: str, max_steps: int = 10):
messages = [
{"role": "system", "content": self.system_prompt},
{"role": "user", "content": task},
]
for step in range(max_steps):
response = await llm.generate(messages, tools=self.tools)
messages.append({"role": "assistant", "content": response})
if response.tool_calls:
for call in response.tool_calls:
result = await self.execute_tool(call)
messages.append({
"role": "tool",
"tool_call_id": call.id,
"content": result
})
else:
# No tool calls - agent is done
return response.content
return "Max steps reached"
async def execute_tool(self, call):
tool = self.tools[call.name]
return await tool.execute(**call.arguments)
Human-in-the-Loop
Pause for human approval at critical checkpoints.
class HumanInLoopAgent(Agent):
def __init__(self, tools, system_prompt, approval_required: list):
super().__init__(tools, system_prompt)
self.approval_required = set(approval_required)
async def execute_tool(self, call):
if call.name in self.approval_required:
approved = await self.request_approval(call)
if not approved:
return "Action cancelled by user"
return await super().execute_tool(call)
async def request_approval(self, call):
print(f"Agent wants to execute: {call.name}({call.arguments})")
response = input("Approve? (y/n): ")
return response.lower() == "y"
ReAct Pattern
ReAct (Reasoning and Acting) alternates between thinking and taking actions.
REACT_PROMPT = """Answer the question using the available tools.
For each step:
1. Thought: Reason about what to do next
2. Action: Choose a tool and inputs
3. Observation: See the result
4. Repeat until you have the answer
Available tools: {tools}
Question: {question}
"""
async def react_agent(question, tools):
prompt = REACT_PROMPT.format(
tools=format_tools(tools),
question=question
)
messages = [{"role": "user", "content": prompt}]
while True:
response = await llm.generate(messages)
messages.append({"role": "assistant", "content": response})
if "Final Answer:" in response:
return extract_final_answer(response)
action = parse_action(response)
if action:
observation = await execute_tool(action, tools)
messages.append({
"role": "user",
"content": f"Observation: {observation}"
})
Advantages:
- Explicit reasoning traces aid debugging
- More interpretable decision-making
- Better handling of complex multi-step tasks
Tool Design
Principles
- Self-contained: Tools return complete, usable information
- Scoped: Each tool does one thing well
- Descriptive: Clear names and descriptions guide the LLM
- Error-robust: Return informative errors, not exceptions
Tool Definition Pattern
class Tool:
def __init__(self, name: str, description: str, parameters: dict, fn):
self.name = name
self.description = description
self.parameters = parameters
self.fn = fn
async def execute(self, **kwargs):
try:
return await self.fn(**kwargs)
except Exception as e:
return f"Error: {str(e)}"
# Example tool
search_tool = Tool(
name="search_database",
description="Search the database for records matching a query. "
"Returns up to 10 matching records with their IDs and summaries.",
parameters={
"query": {"type": "string", "description": "Search query"},
"limit": {"type": "integer", "description": "Max results (default 10)"},
},
fn=search_database
)
Tool Interface Guidelines
- Prefer text inputs/outputs over complex structured data
- Include usage examples in descriptions for ambiguous tools
- Return truncated results when output could be large
- Provide clear feedback on what the tool did
External Context Protocols
Model Context Protocol (MCP) standardizes how agents discover tools, prompts, and resources from external systems. Prefer a protocol boundary when multiple agents, apps, or runtimes need the same integration.
Use MCP-style boundaries when:
- The integration exposes reusable tools or data resources
- You need consistent auth and permission handling outside the agent loop
- Tool metadata should be discoverable rather than hard-coded into prompts
Keep tool schemas tight:
- Use narrow parameters with explicit enums where possible
- Return structured summaries before raw payloads
- Include stable resource IDs so the agent can fetch details on demand
- Keep destructive actions behind explicit approval or dry-run modes
Best Practices
-
Start simple: Begin with the simplest architecture that could work. Add complexity only when it demonstrably improves outcomes.
-
Maintain transparency: Ensure the agent's planning steps are visible. This aids debugging and builds user trust.
-
Design for failure: Agents will make mistakes. Include guardrails, retries, and graceful degradation.
-
Test extensively: Use sandboxed environments. Test edge cases and failure modes, not just happy paths.
-
Limit tool proliferation: More tools means more confusion. Keep the tool set focused and well-documented.
-
Implement checkpoints: For long-running tasks, save state periodically to enable recovery.
-
Set resource limits: Cap iterations, token usage, and tool calls to prevent runaway agents.
-
Log everything: Record all LLM calls, tool executions, and decisions for debugging and improvement.
-
Handle ambiguity: When uncertain, have the agent ask for clarification rather than guessing.
-
Measure outcomes: Track task completion rates, accuracy, and efficiency to guide improvements.
-
Separate integration boundaries: Use MCP or equivalent tool/resource servers when integrations need auth, shared state, or reuse across agents.
References
- Building Effective Agents - Anthropic's guide to agent patterns and best practices
- LangGraph Workflows & Agents - LangGraph documentation on agent architectures
- Model Context Protocol - Standard for exposing tools and resources to agents
- ReAct: Synergizing Reasoning and Acting - Paper introducing the ReAct prompting pattern
Context Engineering
Context engineering is the discipline of curating and maintaining the optimal set of tokens during LLM inference. Unlike prompt engineering (crafting individual prompts), context engineering focuses on what information enters the context window and when.
Table of Contents
- Core Principles
- Context Management Strategies
- System Prompt Design
- Tool Design for Context Efficiency
- Long-Horizon Task Patterns
- Implementation Patterns
- Best Practices
- References
Core Principles
Context as a Finite Resource
LLMs have limited "attention budgets." As context length increases, models experience context rot—decreased ability to accurately recall information. The goal is finding the smallest possible set of high-signal tokens that maximize desired outcomes.
Effective Context = Relevant Information / Total Tokens
Key insight: More context isn't better. The right context is better.
The Context Pollution Problem
Every token added to context has costs:
- Increased latency and compute
- Diluted attention to important information
- Higher risk of hallucination from conflicting data
- Reduced model performance on retrieval tasks
Context Management Strategies
1. Context Trimming
Drop older conversation turns, keeping only the last N turns.
| Aspect | Details |
|---|---|
| Mechanism | Sliding window over conversation history |
| Pros | Deterministic, zero latency, preserves recent context verbatim |
| Cons | Abrupt loss of long-range context, "amnesia" effect |
| Best for | Independent tasks, short interactions, predictable workflows |
def trim_context(messages: list, keep_last_n: int = 10) -> list:
"""Keep system message + last N turns."""
system_msgs = [m for m in messages if m["role"] == "system"]
other_msgs = [m for m in messages if m["role"] != "system"]
return system_msgs + other_msgs[-keep_last_n:]
2. Context Summarization
Compress prior messages into structured summaries.
| Aspect | Details |
|---|---|
| Mechanism | LLM generates summary of older context |
| Pros | Retains long-range memory, smoother UX, scalable |
| Cons | Summarization bias risk, added latency, potential compounding errors |
| Best for | Complex multi-step tasks, long-horizon interactions |
SUMMARIZATION_PROMPT = """Summarize the conversation so far, preserving:
1. Key decisions made
2. Important context established
3. Current task state and goals
4. Any constraints or preferences expressed
Be concise but complete. Output as structured markdown."""
async def summarize_context(messages: list, model) -> str:
"""Generate a summary of conversation history."""
conversation_text = format_messages_for_summary(messages)
response = await model.generate(
system=SUMMARIZATION_PROMPT,
user=conversation_text
)
return response.content
3. Hybrid Approach
Combine trimming and summarization for optimal balance.
class HybridContextManager:
def __init__(
self,
keep_recent: int = 5, # Recent turns to keep verbatim
summary_threshold: int = 20, # When to trigger summarization
):
self.keep_recent = keep_recent
self.summary_threshold = summary_threshold
self.running_summary = ""
def process(self, messages: list) -> list:
if len(messages) < self.summary_threshold:
return messages
# Summarize older messages
old_messages = messages[:-self.keep_recent]
self.running_summary = summarize(old_messages, self.running_summary)
# Return summary + recent messages
return [
{"role": "system", "content": f"Previous context:\n{self.running_summary}"},
*messages[-self.keep_recent:]
]
4. Session Memory
Persist reusable facts, preferences, and task state outside the context window. Load only the relevant slice for the current turn.
| Aspect | Details |
|---|---|
| Mechanism | External store keyed by user, session, task, or resource |
| Pros | Recovers long-range context without carrying all history |
| Cons | Requires retrieval, freshness, and deletion policies |
| Best for | Agents, project work, personalization, long-running workflows |
Separate durable memory from ephemeral scratchpads. Durable memory should contain stable facts and explicit decisions, not every intermediate thought.
System Prompt Design
Principles for Context-Efficient Prompts
- Clear and direct language: Avoid ambiguity that requires clarification turns
- Structured sections: Organize by purpose (role, capabilities, constraints)
- Minimal yet comprehensive: Include only what affects behavior
- Self-contained instructions: Reduce need for context retrieval
Example Structure
# Role
You are [specific role] that [primary function].
# Capabilities
- [Capability 1 with scope]
- [Capability 2 with scope]
# Constraints
- [Hard constraint]
- [Preference]
# Output Format
[Specific format requirements]
Tool Design for Context Efficiency
Just-in-Time Context Loading
Instead of front-loading all possible context, load information dynamically as needed.
# Anti-pattern: Loading everything upfront
context = load_all_user_data() # Large, mostly unused
context += load_all_documents() # Even larger
# Better: Just-in-time retrieval
tools = [
Tool(
name="get_user_preference",
description="Get specific user preference by key",
# Only fetches what's needed when asked
),
Tool(
name="search_documents",
description="Search documents by query",
# Returns relevant subset
),
]
Tool Design Principles
- Self-contained: Each tool returns complete, usable information
- Scoped: Tools do one thing well
- Descriptive: Names and descriptions guide LLM toward correct usage
- Error-robust: Return informative errors that don't pollute context
# Well-designed tool
def search_codebase(query: str, max_results: int = 5) -> str:
"""Search codebase for relevant code snippets.
Args:
query: Natural language description of what to find
max_results: Maximum snippets to return (default 5)
Returns:
Formatted code snippets with file paths and line numbers,
or 'No results found' if nothing matches.
"""
results = perform_search(query, limit=max_results)
if not results:
return "No results found for query."
return format_results(results) # Concise, structured output
Long-Horizon Task Patterns
Pattern 1: Compaction
Periodically compress conversation history to reclaim context space.
async def compaction_loop(agent, messages, task):
while not task.complete:
# Process next step
response = await agent.run(messages)
messages.append(response)
# Compact when approaching limit
if estimate_tokens(messages) > TOKEN_LIMIT * 0.8:
summary = await summarize_context(messages[:-3])
messages = [
{"role": "system", "content": agent.system_prompt},
{"role": "assistant", "content": f"Summary of progress:\n{summary}"},
*messages[-3:] # Keep recent context
]
return messages
Pattern 2: Structured Note-Taking
Agent maintains external notes, retrieving as needed.
class NoteTakingAgent:
def __init__(self):
self.notes = {} # Key-value store outside context
async def run(self, messages):
tools = [
Tool("save_note", self.save_note, "Save information for later"),
Tool("get_note", self.get_note, "Retrieve saved information"),
Tool("list_notes", self.list_notes, "List all saved note keys"),
]
return await self.agent.run(messages, tools=tools)
def save_note(self, key: str, content: str) -> str:
self.notes[key] = content
return f"Saved note: {key}"
def get_note(self, key: str) -> str:
return self.notes.get(key, f"No note found for key: {key}")
Pattern 3: Sub-Agent Architecture
Delegate focused tasks to specialized agents with clean context.
class OrchestratorAgent:
def __init__(self):
self.sub_agents = {
"researcher": ResearchAgent(),
"coder": CodingAgent(),
"reviewer": ReviewAgent(),
}
async def delegate(self, task: str, agent_type: str) -> str:
"""Delegate to sub-agent, receive condensed summary."""
agent = self.sub_agents[agent_type]
# Sub-agent works with fresh context
result = await agent.run(task)
# Return only essential findings to main context
return result.summary # Not the full conversation
Benefits:
- Each sub-agent has focused, clean context
- Main agent receives condensed results
- Parallelization opportunities
- Failure isolation
Implementation Patterns
Session Memory Manager
class SessionMemory:
def __init__(
self,
keep_last_n_turns: int = 5,
context_limit: int = 100_000, # tokens
summarizer = None,
):
self.keep_last_n_turns = keep_last_n_turns
self.context_limit = context_limit
self.summarizer = summarizer
self.messages = []
self.summary = ""
async def add_message(self, message: dict):
self.messages.append(message)
await self._maybe_compact()
async def _maybe_compact(self):
current_tokens = estimate_tokens(self.messages)
if current_tokens > self.context_limit * 0.8:
# Summarize all but recent messages
old_messages = self.messages[:-self.keep_last_n_turns]
new_summary = await self.summarizer.summarize(
old_messages,
previous_summary=self.summary
)
self.summary = new_summary
self.messages = self.messages[-self.keep_last_n_turns:]
def get_context(self) -> list:
context = []
if self.summary:
context.append({
"role": "system",
"content": f"Conversation summary:\n{self.summary}"
})
context.extend(self.messages)
return context
Token Estimation
def estimate_tokens(messages: list) -> int:
"""Rough token estimation (4 chars ≈ 1 token for English)."""
total_chars = sum(
len(m.get("content", ""))
for m in messages
)
return total_chars // 4
def estimate_tokens_accurate(messages: list, model: str) -> int:
"""Accurate token count using tiktoken."""
import tiktoken
encoding = tiktoken.encoding_for_model(model)
return sum(
len(encoding.encode(m.get("content", "")))
for m in messages
)
Best Practices
-
Treat context as precious: Every token has a cost. Include only information that improves task performance.
-
Use progressive disclosure: Start minimal, expand context only when needed via tools.
-
Design for recoverability: Agents should be able to reconstruct critical context from external sources.
-
Monitor context health: Track token usage, retrieval accuracy, and task completion rates.
-
Prefer structured over raw data: JSON, markdown tables, and clear formatting improve information density.
-
Implement graceful degradation: When context limits approach, prioritize recent and high-signal information.
-
Test with long conversations: Validate agent behavior after many turns, not just initial interactions.
-
Separate concerns: Use different context regions for system instructions, user history, and tool outputs.
-
Version your summaries: When compacting, maintain enough structure to debug summarization issues.
-
Measure and iterate: Context engineering is empirical—test what information actually improves outcomes.
-
Separate memory classes: Keep instructions, recent messages, durable memory, retrieved documents, and tool results in distinct sections so conflicts are easier to detect.
References
- reference/evaluation-strategies.md - Testing context management effectiveness
- reference/summarization-patterns.md - Detailed summarization implementations
- Effective Context Engineering for AI Agents
- OpenAI Session Memory for Agents
Using LoRA for Fine-tuning
LoRA (Low-Rank Adaptation) enables efficient fine-tuning by freezing pretrained weights and injecting small trainable matrices into transformer layers. This reduces trainable parameters to ~0.1% of the original model while maintaining performance.
Table of Contents
- Core Concepts
- Basic Setup
- Configuration Parameters
- QLoRA (Quantized LoRA)
- Training Patterns
- Saving and Loading
- Merging Adapters
- Best Practices
- References
Core Concepts
How LoRA Works
Instead of updating all weights during fine-tuning, LoRA decomposes weight updates into low-rank matrices:
W' = W + BA
Where:
Wis the frozen pretrained weight matrix (d × k)Bis a trainable matrix (d × r)Ais a trainable matrix (r × k)ris the rank, much smaller than d and k
The key insight: weight updates during fine-tuning have low intrinsic rank, so we can represent them efficiently with smaller matrices.
Why Use LoRA
| Aspect | Full Fine-tuning | LoRA |
|---|---|---|
| Trainable params | 100% | ~0.1-1% |
| Memory usage | High | Low |
| Adapter size | Full model | ~3-100 MB |
| Training speed | Slower | Faster |
| Multiple tasks | Separate models | Swap adapters |
Basic Setup
Installation
pip install peft transformers accelerate
Minimal Example
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model, TaskType
import torch
# Load base model
model_name = "meta-llama/Llama-3.2-1B"
model = AutoModelForCausalLM.from_pretrained(
model_name,
dtype=torch.bfloat16,
device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token
# Configure LoRA
lora_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
lora_dropout=0.05,
bias="none",
task_type=TaskType.CAUSAL_LM,
)
# Apply LoRA
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 3,407,872 || all params: 1,238,300,672 || trainable%: 0.28%
Configuration Parameters
LoraConfig Options
from peft import LoraConfig, TaskType
config = LoraConfig(
# Core parameters
r=16, # Rank of update matrices
lora_alpha=32, # Scaling factor (alpha/r applied to updates)
target_modules=["q_proj", "v_proj"], # Layers to adapt
# Regularization
lora_dropout=0.05, # Dropout on LoRA layers
bias="none", # "none", "all", or "lora_only"
# Task configuration
task_type=TaskType.CAUSAL_LM, # CAUSAL_LM, SEQ_CLS, SEQ_2_SEQ_LM, TOKEN_CLS
# Advanced
modules_to_save=None, # Additional modules to train (e.g., ["lm_head"])
layers_to_transform=None, # Specific layer indices to adapt
rank_pattern=None, # Per-module rank overrides
alpha_pattern=None, # Per-module alpha overrides
trainable_token_indices=None, # Train only selected new token embeddings
target_parameters=None, # Target nn.Parameter weights in some MoE layers
use_rslora=False, # Rank-stabilized LoRA scaling
use_dora=False, # Weight-Decomposed LoRA
)
Target Modules by Architecture
# Llama, Mistral, Qwen
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
# GPT-2, GPT-J
target_modules = ["c_attn", "c_proj", "c_fc"]
# BERT, RoBERTa
target_modules = ["query", "key", "value", "dense"]
# Falcon
target_modules = ["query_key_value", "dense", "dense_h_to_4h", "dense_4h_to_h"]
# Phi
target_modules = ["q_proj", "k_proj", "v_proj", "dense", "fc1", "fc2"]
# MoE expert parameters stored as nn.Parameter tensors
target_parameters = ["feed_forward.experts.gate_up_proj", "feed_forward.experts.down_proj"]
For MoE models where expert weights are not nn.Linear modules, use target_parameters rather than target_modules. Merge adapters before latency-sensitive inference because materializing expert LoRA updates can add overhead.
Finding Target Modules
# Print all linear layer names
from peft.utils import get_peft_model_state_dict
def find_target_modules(model):
linear_modules = set()
for name, module in model.named_modules():
if isinstance(module, torch.nn.Linear):
# Get the last part of the name (e.g., "q_proj" from "model.layers.0.self_attn.q_proj")
layer_name = name.split(".")[-1]
linear_modules.add(layer_name)
return list(linear_modules)
print(find_target_modules(model))
QLoRA (Quantized LoRA)
QLoRA combines 4-bit quantization with LoRA, enabling fine-tuning of large models on consumer GPUs.
Setup
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
import torch
# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4", # Normalized float 4-bit
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True, # Nested quantization
)
# Load quantized model
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.2-3B",
quantization_config=bnb_config,
device_map="auto",
)
# Prepare for k-bit training
model = prepare_model_for_kbit_training(model)
# Apply LoRA
lora_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
lora_dropout=0.05,
bias="none",
task_type=TaskType.CAUSAL_LM,
)
model = get_peft_model(model, lora_config)
Memory Requirements
| Model Size | Full FT (16-bit) | LoRA (16-bit) | QLoRA (4-bit) |
|---|---|---|---|
| 7B | ~60 GB | ~16 GB | ~6 GB |
| 13B | ~104 GB | ~28 GB | ~10 GB |
| 70B | ~560 GB | ~160 GB | ~48 GB |
Training Patterns
With Hugging Face Trainer
from transformers import TrainingArguments, Trainer, DataCollatorForLanguageModeling
from datasets import load_dataset
# Prepare dataset
dataset = load_dataset("tatsu-lab/alpaca", split="train")
def format_prompt(example):
if example["input"]:
text = f"### Instruction:\n{example['instruction']}\n\n### Input:\n{example['input']}\n\n### Response:\n{example['output']}"
else:
text = f"### Instruction:\n{example['instruction']}\n\n### Response:\n{example['output']}"
return {"text": text}
dataset = dataset.map(format_prompt)
def tokenize(examples):
return tokenizer(
examples["text"],
truncation=True,
max_length=512,
padding=False,
)
tokenized = dataset.map(tokenize, batched=True, remove_columns=dataset.column_names)
# Training arguments (note higher learning rate)
training_args = TrainingArguments(
output_dir="./lora-output",
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
num_train_epochs=1,
learning_rate=2e-4, # Higher than full fine-tuning
bf16=True,
logging_steps=10,
save_steps=500,
warmup_ratio=0.03,
gradient_checkpointing=True,
optim="adamw_torch_fused",
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=tokenized,
data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False),
)
trainer.train()
With SFTTrainer (TRL)
from trl import SFTTrainer, SFTConfig
sft_config = SFTConfig(
output_dir="./sft-lora",
max_length=1024,
dataset_text_field="text",
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
num_train_epochs=1,
learning_rate=2e-4,
bf16=True,
logging_steps=10,
gradient_checkpointing=True,
)
trainer = SFTTrainer(
model=model,
args=sft_config,
train_dataset=dataset,
processing_class=tokenizer,
peft_config=lora_config, # Pass config directly, SFTTrainer applies it
)
trainer.train()
Classification Task
from transformers import AutoModelForSequenceClassification
from peft import LoraConfig, get_peft_model, TaskType
model = AutoModelForSequenceClassification.from_pretrained(
"bert-base-uncased",
num_labels=2,
)
lora_config = LoraConfig(
r=8,
lora_alpha=16,
target_modules=["query", "value"],
lora_dropout=0.1,
bias="none",
task_type=TaskType.SEQ_CLS,
modules_to_save=["classifier"], # Train classification head fully
)
model = get_peft_model(model, lora_config)
Saving and Loading
Save Adapter
# Save only LoRA weights (small file)
model.save_pretrained("./my-lora-adapter")
tokenizer.save_pretrained("./my-lora-adapter")
# Push to Hub
model.push_to_hub("username/my-lora-adapter")
Load Adapter
from peft import PeftModel
from transformers import AutoModelForCausalLM
# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.2-1B",
dtype=torch.bfloat16,
device_map="auto",
)
# Load adapter
model = PeftModel.from_pretrained(base_model, "./my-lora-adapter")
# For inference
model.eval()
Switch Between Adapters
# Load multiple adapters
model.load_adapter("./adapter-1", adapter_name="task1")
model.load_adapter("./adapter-2", adapter_name="task2")
# Switch active adapter
model.set_adapter("task1")
output = model.generate(**inputs)
model.set_adapter("task2")
output = model.generate(**inputs)
# Disable adapter (use base model)
with model.disable_adapter():
output = model.generate(**inputs)
Per-Sample Adapter Selection
For inference batches that mix tasks or languages, pass adapter_names aligned with each sample. Use "__base__" for rows that should run the base model without an adapter.
inputs = tokenizer(["Hello", "Bonjour", "Hallo"], return_tensors="pt", padding=True).to(model.device)
adapter_names = ["__base__", "french", "german"]
outputs = model.generate(**inputs, adapter_names=adapter_names, max_new_tokens=32)
Merging Adapters
Merge LoRA weights into the base model for deployment without adapter overhead.
from peft import PeftModel
# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.2-1B",
dtype=torch.bfloat16,
device_map="cpu", # Merge on CPU to avoid memory issues
)
# Load adapter
model = PeftModel.from_pretrained(base_model, "./my-lora-adapter")
# Merge and unload
merged_model = model.merge_and_unload()
# Save merged model
merged_model.save_pretrained("./merged-model")
tokenizer.save_pretrained("./merged-model")
# Push merged model to Hub
merged_model.push_to_hub("username/my-merged-model")
Best Practices
-
Start with r=16: Scale up to 32 or 64 if the model underfits, down to 8 if overfitting or memory-constrained
-
Set lora_alpha = 2 × r: This is a common heuristic; the effective scaling is
alpha/r -
Target all attention and MLP layers: For best results on LLMs, include gate/up/down projections:
target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"] -
Use higher learning rate: 2e-4 is typical for LoRA vs 2e-5 for full fine-tuning
-
Enable gradient checkpointing: Reduces memory at cost of ~20% slower training:
model.gradient_checkpointing_enable() -
Use QLoRA for large models: Essential for fine-tuning 7B+ models on consumer GPUs
-
Keep dropout low: 0.05 is usually sufficient; higher values may hurt performance
-
Save checkpoints frequently: LoRA adapters are small, so save often
-
Evaluate on base model too: Ensure adapter doesn't degrade base capabilities
-
Consider modules_to_save for task heads: For classification, train the classifier fully:
modules_to_save=["classifier", "score"] -
Train new tokens selectively: After adding special tokens, prefer
trainable_token_indicesover training the whole embedding matrix when only a few tokens need adaptation. -
Merge adapters for high-throughput serving:
merge_and_unload()removes PEFT runtime overhead, especially for MoE parameter-targeted adapters.
References
See reference/ for detailed documentation:
advanced-techniques.md- DoRA, rsLoRA, adapter composition, and debugging
External documentation:
Using MLX for LLMs on Apple Silicon
MLX-LM is a Python package for running large language models on Apple Silicon, leveraging the MLX framework for optimized performance with unified memory architecture.
Table of Contents
- Core Concepts
- Installation
- Text Generation
- Interactive Chat
- Model Conversion
- Quantization
- Fine-tuning with LoRA
- Serving Models
- Best Practices
- References
Core Concepts
Why MLX
| Aspect | PyTorch on Mac | MLX |
|---|---|---|
| Memory | Separate CPU/GPU copies | Unified memory, no copies |
| Optimization | Generic Metal backend | Apple Silicon native |
| Model loading | Slower, more memory | Lazy loading, efficient |
| Quantization | Limited support | Built-in 4/8-bit |
MLX arrays live in shared memory, accessible by both CPU and GPU without data transfer overhead.
Supported Models
MLX-LM supports most popular architectures: Llama, Mistral, Qwen, Phi, Gemma, Cohere, and many more. Check the mlx-community on Hugging Face for pre-converted models.
Installation
pip install mlx-lm
Requires macOS 13.5+ and Apple Silicon (M1/M2/M3/M4).
Text Generation
Python API
from mlx_lm import load, generate
# Load model (from HF hub or local path)
model, tokenizer = load("mlx-community/Llama-3.2-3B-Instruct-4bit")
# Generate text
response = generate(
model,
tokenizer,
prompt="Explain quantum computing in simple terms:",
max_tokens=256,
temp=0.7,
)
print(response)
Streaming Generation
from mlx_lm import load, stream_generate
model, tokenizer = load("mlx-community/Mistral-7B-Instruct-v0.3-4bit")
prompt = "Write a haiku about programming:"
for response in stream_generate(model, tokenizer, prompt, max_tokens=100):
print(response.text, end="", flush=True)
print()
Batch Generation
from mlx_lm import load, batch_generate
model, tokenizer = load("mlx-community/Qwen2.5-7B-Instruct-4bit")
prompts = [
"What is machine learning?",
"Explain neural networks:",
"Define deep learning:",
]
responses = batch_generate(
model,
tokenizer,
prompts,
max_tokens=100,
)
for prompt, response in zip(prompts, responses):
print(f"Q: {prompt}\nA: {response}\n")
CLI Generation
# Basic generation
mlx_lm.generate --model mlx-community/Llama-3.2-3B-Instruct-4bit \
--prompt "Explain recursion:" \
--max-tokens 256
# With sampling parameters
mlx_lm.generate --model mlx-community/Mistral-7B-Instruct-v0.3-4bit \
--prompt "Write a poem about AI:" \
--temp 0.8 \
--top-p 0.95
Interactive Chat
CLI Chat
# Start chat REPL (context preserved between turns)
mlx_lm.chat --model mlx-community/Llama-3.2-3B-Instruct-4bit
Python Chat
from mlx_lm import load, generate
model, tokenizer = load("mlx-community/Llama-3.2-3B-Instruct-4bit")
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What's the capital of France?"},
]
prompt = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
response = generate(model, tokenizer, prompt=prompt, max_tokens=256)
print(response)
Model Conversion
Convert Hugging Face models to MLX format:
CLI Conversion
# Convert with 4-bit quantization
mlx_lm.convert --model meta-llama/Llama-3.2-3B-Instruct \
-q # Quantize to 4-bit
# With specific quantization
mlx_lm.convert --model mistralai/Mistral-7B-Instruct-v0.3 \
-q \
--q-bits 8 \
--q-group-size 64
# Upload to Hugging Face Hub
mlx_lm.convert --model meta-llama/Llama-3.2-1B-Instruct \
-q \
--upload-repo your-username/Llama-3.2-1B-Instruct-4bit-mlx
Python Conversion
from mlx_lm import convert
repo = "meta-llama/Llama-3.2-3B-Instruct"
convert(
repo,
quantize=True,
mlx_path="./llama-3.2-3b-mlx",
q_bits=4,
q_group_size=64,
)
Conversion Options
| Option | Default | Description |
|---|---|---|
--q-bits | 4 | Quantization bits (4 or 8) |
--q-group-size | 64 | Group size for quantization |
--dtype | float16 | Data type for non-quantized weights |
Quantization
MLX supports multiple quantization methods for different use cases:
| Method | Best For | Command |
|---|---|---|
| Basic | Quick conversion | mlx_lm.convert -q |
| DWQ | Quality-preserving | mlx_lm.dwq |
| AWQ | Activation-aware | mlx_lm.awq |
| Dynamic | Per-layer precision | mlx_lm.dynamic_quant |
| GPTQ | Established method | mlx_lm.gptq |
Quick Quantization
# 4-bit quantization during conversion
mlx_lm.convert --model mistralai/Mistral-7B-v0.3 -q
# 8-bit for higher quality
mlx_lm.convert --model mistralai/Mistral-7B-v0.3 -q --q-bits 8
For detailed coverage of each method, see reference/quantization.md.
Fine-tuning with LoRA
MLX supports LoRA and QLoRA fine-tuning for efficient adaptation on Apple Silicon.
Quick Start
pip install "mlx-lm[train]"
# Prepare training data (JSONL format)
# ./data/train.jsonl is required for --train
# ./data/valid.jsonl is optional
# Supported rows include {"text": "..."} and chat {"messages": [...]}
# Fine-tune with LoRA
mlx_lm.lora --model mlx-community/Llama-3.2-3B-Instruct-4bit \
--train \
--data ./data \
--iters 1000
# Completion-only loss for chat/completion datasets
mlx_lm.lora --model mlx-community/Llama-3.2-3B-Instruct-4bit \
--train \
--data ./data \
--mask-prompt
# Generate with adapter
mlx_lm.generate --model mlx-community/Llama-3.2-3B-Instruct-4bit \
--adapter-path ./adapters \
--prompt "Your prompt here"
Fuse Adapter into Model
# Merge LoRA weights into base model
mlx_lm.fuse --model mlx-community/Llama-3.2-3B-Instruct-4bit \
--adapter-path ./adapters \
--save-path ./fused-model
# Or export to GGUF
mlx_lm.fuse --model mlx-community/Llama-3.2-3B-Instruct-4bit \
--adapter-path ./adapters \
--export-gguf
For detailed LoRA configuration and training patterns, see reference/fine-tuning.md.
MLX-LM also supports --fine-tune-type dora and --fine-tune-type full when the task needs more capacity than standard LoRA.
Serving Models
OpenAI-Compatible Server
# Start server
mlx_lm.server --model mlx-community/Llama-3.2-3B-Instruct-4bit --port 8080
# Use with OpenAI client
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "default",
"messages": [{"role": "user", "content": "Hello!"}],
"max_tokens": 256
}'
Python Client
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
response = client.chat.completions.create(
model="default",
messages=[{"role": "user", "content": "Explain MLX in one sentence."}],
max_tokens=100,
)
print(response.choices[0].message.content)
Best Practices
-
Use pre-quantized models: Download from
mlx-communityon Hugging Face for immediate use -
Match quantization to your hardware: M1/M2 with 8GB: use 4-bit; M2/M3 Pro/Max: 8-bit for quality
-
Leverage unified memory: Unlike CUDA, MLX models can exceed "GPU memory" by using swap (slower but works)
-
Use streaming for UX:
stream_generateprovides responsive output for interactive applications -
Cache prompt prefixes: Use
mlx_lm.cache_promptfor repeated prompts with varying suffixes -
Limit KV cache for long generations: Use
--max-kv-sizeto cap memory at the cost of long-range quality -
Batch similar requests:
batch_generateis more efficient than sequential generation -
Start with 4-bit quantization: Good quality/size tradeoff; upgrade to 8-bit if quality issues
-
Fuse adapters for deployment: After fine-tuning, fuse adapters for faster inference without loading separately
-
Monitor memory with Activity Monitor: Watch memory pressure to avoid swap thrashing
-
Use chat templates: Always apply
tokenizer.apply_chat_template()for instruction-tuned models
References
See reference/ for detailed documentation:
quantization.md- Detailed quantization methods and when to use eachfine-tuning.md- Complete LoRA/QLoRA training guide with data formats and configuration
External documentation:
Prompt Engineering
Prompt engineering is the practice of designing inputs that guide LLMs to produce desired outputs. Effective prompts reduce errors, improve consistency, and unlock model capabilities.
Table of Contents
- Core Principles
- Be Clear and Direct
- Use Examples (Multishot)
- Reasoning Guidance
- XML Tags
- Role Prompting
- Long Context
- Output Control
- Self-Verification
- Best Practices
- References
Core Principles
Golden rule: Show your prompt to a colleague with minimal context. If they're confused, the model will be too.
- Be explicit - State exactly what you want; never assume the model knows your preferences
- Provide context - Include what the output is for, who the audience is, and what success looks like
- Use structure - Sequential steps, XML tags, and clear formatting reduce ambiguity
- Show examples - Demonstrations outperform descriptions for complex formats
Be Clear and Direct
Treat the model as a capable but context-free collaborator. Specify:
- What the task results will be used for
- What audience the output is meant for
- What a successful completion looks like
Vague vs Specific
# Vague
Analyze this data and give insights.
# Specific
Analyze this Q2 sales data for our board presentation.
1. Identify the top 3 revenue trends
2. Flag any anomalies exceeding 15% variance
3. Recommend 2-3 actionable next steps
Format as bullet points, max 200 words.
Sequential Steps
Use numbered lists for multi-step tasks:
Your task is to anonymize customer feedback.
Instructions:
1. Replace customer names with "CUSTOMER_[ID]"
2. Replace emails with "EMAIL_[ID]@example.com"
3. Redact phone numbers as "PHONE_[ID]"
4. Leave product names intact
5. Output only processed messages, separated by "---"
Use Examples (Multishot)
Provide 3-5 diverse examples to demonstrate expected behavior. Examples reduce misinterpretation and enforce consistent formatting.
Structure
Categorize customer feedback by issue type and sentiment.
<examples>
<example>
Input: The dashboard loads slowly and the export button is hidden.
Category: UI/UX, Performance
Sentiment: Negative
Priority: High
</example>
<example>
Input: Love the Salesforce integration! Would be great to add Hubspot.
Category: Integration, Feature Request
Sentiment: Positive
Priority: Medium
</example>
</examples>
Now categorize: {{FEEDBACK}}
Tips
- Make examples relevant to actual use cases
- Include edge cases and potential challenges
- Vary examples to prevent unintended pattern matching
- Wrap in
<example>tags for clarity
Reasoning Guidance
For complex tasks, guide the model to reason before answering, but ask for concise conclusions or verifiable work rather than unrestricted hidden chain-of-thought.
Basic
Determine the best investment option for this client. Think step-by-step.
Guided
Specify what steps to consider:
Think before answering:
1. Consider the client's risk tolerance given their 5-year timeline
2. Calculate potential returns for each option
3. Factor in market volatility history
4. Then provide your recommendation
Structured (Recommended)
Separate private analysis instructions from the required answer format:
Analyze this contract for legal risks.
Before answering, check:
- Indemnification implications
- Liability exposure
- IP ownership concerns
Then provide:
<answer>
- Top risks
- Recommended edits
- Open questions
</answer>
When you need auditability, ask for brief supporting rationale, calculations, cited evidence, or a checklist of checks performed. Do not require full chain-of-thought unless the target model or product explicitly supports exposing it.
XML Tags
Use XML tags to separate prompt components. This prevents instruction/content confusion and improves parseability.
Common Tags
<instructions>Task steps and requirements</instructions>
<context>Background information</context>
<document>Source material to process</document>
<example>Demonstration of expected behavior</example>
<constraints>Boundaries and limitations</constraints>
<output_format>Expected response structure</output_format>
Nested Structure
<documents>
<document index="1">
<source>annual_report_2023.pdf</source>
<content>{{REPORT_CONTENT}}</content>
</document>
<document index="2">
<source>competitor_analysis.xlsx</source>
<content>{{ANALYSIS_CONTENT}}</content>
</document>
</documents>
<instructions>
Compare revenue trends across both documents.
Identify strategic advantages mentioned in the annual report.
</instructions>
Reference Tags in Instructions
Be explicit when referring to tagged content:
Using the contract in <contract> tags, identify all clauses
related to termination.
Role Prompting
Set expertise context via system prompts to improve domain-specific performance.
System Prompt Pattern
system = "You are a senior securities lawyer at a Fortune 500 company."
user = "Review this acquisition agreement for regulatory risks."
Effective Roles
# General
You are a [role] at [organization type].
# Specific (better)
You are the General Counsel of a Fortune 500 tech company
specializing in M&A transactions.
# With behavioral guidance (best)
You are a senior data scientist. You prioritize statistical
rigor over speed. When uncertain, you state assumptions
explicitly and suggest validation approaches.
When to Use
- Complex analysis requiring domain expertise
- Tasks where tone/style matters (legal, medical, executive)
- When a specific perspective would improve output quality
Long Context
For prompts with large documents (20K+ tokens):
Document Placement
Place long documents at the top, before instructions:
<documents>
{{LARGE_DOCUMENT_CONTENT}}
</documents>
<instructions>
Summarize the key findings from the document above.
Focus on financial implications.
</instructions>
Quote Grounding
Ask the model to cite sources before analyzing:
<documents>
{{PATIENT_RECORDS}}
</documents>
First, find and quote the relevant sections in <quotes> tags.
Then provide your diagnosis in <analysis> tags, referencing
the quoted evidence.
Multi-Document Metadata
Include source information for attribution:
<documents>
<document index="1">
<source>quarterly_report_q2.pdf</source>
<date>2024-07-15</date>
<content>{{CONTENT}}</content>
</document>
</documents>
Output Control
Verbosity Specification
<output_format>
- Default responses: 3-6 sentences or ≤5 bullets
- Simple factual questions: ≤2 sentences
- Complex analysis: 1 overview paragraph + ≤5 tagged bullets
</output_format>
Format Constraints
Output requirements:
- Use markdown tables for comparisons
- Code blocks for any technical content
- No introductory phrases ("Here's...", "Sure...")
- End with exactly 3 action items
Scope Boundaries
Prevent drift from original intent:
Implement EXACTLY and ONLY what is requested.
- Do not add features beyond the specification
- Do not refactor surrounding code
- Choose the simplest valid interpretation
- Ask for clarification rather than assuming
Self-Verification
For high-stakes outputs, include verification steps:
<verification>
Before finalizing your response:
1. Re-read the original request
2. Check that all requirements are addressed
3. Verify any specific claims against provided documents
4. Soften language where certainty is low
5. Flag any assumptions you made
</verification>
Uncertainty Acknowledgment
When uncertain:
- Explicitly state "Based on the provided context..."
- Offer 2-3 plausible interpretations if ambiguous
- Never fabricate specific details (dates, numbers, quotes)
- Say "I don't have enough information to..." when applicable
Best Practices
- Start specific, then generalize - Begin with detailed prompts; relax constraints only after validating output quality
- Test with edge cases - Include unusual inputs in your evaluation to catch failure modes
- Iterate on examples - When outputs miss the mark, add an example demonstrating the correct behavior
- Separate instructions from content - Use XML tags to prevent the model from confusing your instructions with input data
- Put documents before queries - For long context, place source material at the top of the prompt
- Ask for verifiable reasoning artifacts - Prefer concise rationale, calculations, citations, or checklists over full hidden chain-of-thought
- Constrain output format explicitly - Specify structure, length, and style to reduce post-processing
- Version your prompts - Track changes to understand what modifications improved or degraded performance
- Use system prompts for role, user prompts for task - Keep role context stable; vary task instructions
- Validate with fresh eyes - Have someone unfamiliar with the task review your prompt for clarity
References
Using PyTorch
PyTorch is a deep learning framework with dynamic computation graphs, strong GPU acceleration, and Pythonic design. This skill covers practical patterns for building production-quality neural networks.
Table of Contents
- Core Concepts
- Model Architecture
- Training Loop
- Data Loading
- Performance Optimization
- Distributed Training
- Saving and Loading
- Best Practices
- References
Core Concepts
Tensors
import torch
# Create tensors
x = torch.tensor([[1, 2], [3, 4]], dtype=torch.float32)
x = torch.zeros(3, 4)
x = torch.randn(3, 4) # Normal distribution
# Device management
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
x = x.to(device)
# Operations (all return new tensors)
y = x + 1
y = x @ x.T # Matrix multiplication
y = x.view(2, 6) # Reshape
Autograd
# Enable gradient tracking
x = torch.randn(3, requires_grad=True)
y = x ** 2
loss = y.sum()
# Compute gradients
loss.backward()
print(x.grad) # dy/dx
# Disable gradients for inference
with torch.no_grad():
pred = model(x)
# Or use inference mode (more efficient)
with torch.inference_mode():
pred = model(x)
Model Architecture
nn.Module Pattern
import torch.nn as nn
import torch.nn.functional as F
class Model(nn.Module):
def __init__(self, input_dim: int, hidden_dim: int, output_dim: int):
super().__init__()
self.fc1 = nn.Linear(input_dim, hidden_dim)
self.fc2 = nn.Linear(hidden_dim, output_dim)
self.dropout = nn.Dropout(0.1)
def forward(self, x: torch.Tensor) -> torch.Tensor:
x = F.relu(self.fc1(x))
x = self.dropout(x)
return self.fc2(x)
Common Layers
# Convolution
nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1)
# Normalization
nn.BatchNorm2d(num_features)
nn.LayerNorm(normalized_shape)
# Attention
nn.MultiheadAttention(embed_dim, num_heads)
# Recurrent
nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)
nn.GRU(input_size, hidden_size, num_layers, batch_first=True)
Weight Initialization
def init_weights(module):
if isinstance(module, nn.Linear):
nn.init.xavier_uniform_(module.weight)
if module.bias is not None:
nn.init.zeros_(module.bias)
elif isinstance(module, nn.Embedding):
nn.init.normal_(module.weight, std=0.02)
model.apply(init_weights)
Training Loop
Standard Pattern
model = Model(input_dim, hidden_dim, output_dim).to(device)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)
criterion = nn.CrossEntropyLoss()
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_epochs)
for epoch in range(num_epochs):
model.train()
for batch in train_loader:
inputs, targets = batch
inputs, targets = inputs.to(device), targets.to(device)
optimizer.zero_grad()
outputs = model(inputs)
loss = criterion(outputs, targets)
loss.backward()
# Optional: gradient clipping
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()
scheduler.step()
# Validation
model.eval()
with torch.no_grad():
for batch in val_loader:
# ... validation logic
Mixed Precision Training
from torch.amp import autocast, GradScaler
scaler = GradScaler("cuda")
for batch in train_loader:
inputs, targets = batch
inputs, targets = inputs.to(device), targets.to(device)
optimizer.zero_grad()
with autocast("cuda", dtype=torch.bfloat16):
outputs = model(inputs)
loss = criterion(outputs, targets)
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()
Gradient Accumulation
# Requires setup from Mixed Precision Training above:
# scaler = GradScaler(), model, criterion, optimizer, device
accumulation_steps = 4
for i, batch in enumerate(train_loader):
inputs, targets = batch
inputs, targets = inputs.to(device), targets.to(device)
with autocast("cuda", dtype=torch.bfloat16):
outputs = model(inputs)
loss = criterion(outputs, targets) / accumulation_steps
scaler.scale(loss).backward()
if (i + 1) % accumulation_steps == 0:
scaler.step(optimizer)
scaler.update()
optimizer.zero_grad()
Data Loading
Dataset and DataLoader
from torch.utils.data import Dataset, DataLoader
class CustomDataset(Dataset):
def __init__(self, data, labels, transform=None):
self.data = data
self.labels = labels
self.transform = transform
def __len__(self):
return len(self.data)
def __getitem__(self, idx):
x = self.data[idx]
if self.transform:
x = self.transform(x)
return x, self.labels[idx]
train_loader = DataLoader(
dataset,
batch_size=32,
shuffle=True,
num_workers=4,
pin_memory=True, # Faster GPU transfer
drop_last=True, # Consistent batch sizes
)
Collate Functions
def collate_fn(batch):
"""Custom batching for variable-length sequences."""
inputs, targets = zip(*batch)
inputs = nn.utils.rnn.pad_sequence(inputs, batch_first=True)
targets = torch.stack(targets)
return inputs, targets
loader = DataLoader(dataset, collate_fn=collate_fn)
Performance Optimization
torch.compile (PyTorch 2.0+)
# Basic compilation
model = torch.compile(model)
# With options
model = torch.compile(
model,
mode="reduce-overhead", # Options: default, reduce-overhead, max-autotune
fullgraph=True, # Enforce no graph breaks
)
# Compile specific functions
@torch.compile
def train_step(model, inputs, targets):
outputs = model(inputs)
return criterion(outputs, targets)
Compilation modes:
default: Good balance of compile time and speedupreduce-overhead: Minimizes framework overhead, good for small modelsmax-autotune: Maximum performance, longer compile time
Start with fullgraph=False unless you are actively fixing graph breaks. Use TORCH_LOGS="graph_breaks" to inspect why compilation falls back to eager execution.
Memory Optimization
# Activation checkpointing (trade compute for memory)
from torch.utils.checkpoint import checkpoint
class Model(nn.Module):
def forward(self, x):
# Recompute activations during backward
x = checkpoint(self.expensive_layer, x, use_reentrant=False)
return self.output_layer(x)
# Clear cache
torch.cuda.empty_cache()
# Monitor memory
print(torch.cuda.memory_allocated() / 1e9, "GB")
print(torch.cuda.max_memory_allocated() / 1e9, "GB")
Distributed Training
DistributedDataParallel (DDP)
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data.distributed import DistributedSampler
def setup(rank, world_size):
dist.init_process_group("nccl", rank=rank, world_size=world_size)
torch.cuda.set_device(rank)
def cleanup():
dist.destroy_process_group()
def train(rank, world_size):
setup(rank, world_size)
model = Model().to(rank)
model = DDP(model, device_ids=[rank])
sampler = DistributedSampler(dataset, num_replicas=world_size, rank=rank)
loader = DataLoader(dataset, sampler=sampler)
for epoch in range(num_epochs):
sampler.set_epoch(epoch) # Important for shuffling
# ... training loop
cleanup()
# Launch with: torchrun --nproc_per_node=4 train.py
FullyShardedDataParallel (FSDP)
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
from torch.distributed.fsdp import MixedPrecision
mp_policy = MixedPrecision(
param_dtype=torch.bfloat16,
reduce_dtype=torch.bfloat16,
buffer_dtype=torch.bfloat16,
)
model = FSDP(
model,
mixed_precision=mp_policy,
use_orig_params=True, # Required for torch.compile compatibility
)
Saving and Loading
Checkpoints
# Save
torch.save({
"epoch": epoch,
"model_state_dict": model.state_dict(),
"optimizer_state_dict": optimizer.state_dict(),
"loss": loss,
}, "checkpoint.pt")
# Load
checkpoint = torch.load("checkpoint.pt", map_location=device, weights_only=True)
model.load_state_dict(checkpoint["model_state_dict"])
optimizer.load_state_dict(checkpoint["optimizer_state_dict"])
Export for Deployment
# TorchScript
scripted = torch.jit.script(model)
scripted.save("model.pt")
# ONNX
torch.onnx.export(
model,
dummy_input,
"model.onnx",
input_names=["input"],
output_names=["output"],
dynamic_axes={"input": {0: "batch"}, "output": {0: "batch"}},
)
Best Practices
-
Always set model mode: Use
model.train()andmodel.eval()appropriately -
Use inference_mode over no_grad: More efficient for inference
-
Pin memory for GPU training: Set
pin_memory=Truein DataLoader -
Profile before optimizing: Use
torch.profilerto find bottlenecks -
Prefer bfloat16 over float16: Better numerical stability on modern GPUs
-
Use torch.compile: Significant speedups with minimal code changes
-
Set deterministic mode for reproducibility:
torch.manual_seed(42) torch.use_deterministic_algorithms(True) torch.backends.cudnn.deterministic = True torch.backends.cudnn.benchmark = False -
Use the unified AMP API: Prefer
torch.amp.autocast("cuda")andtorch.amp.GradScaler("cuda")over legacytorch.cuda.amp. -
Load untrusted checkpoints cautiously: Use
weights_only=Truewhen loading state-dict checkpoints.
References
See reference/ for detailed documentation:
training-patterns.md- Advanced training techniquesdebugging.md- Debugging and profiling tools
External documentation:
QLoRA: Quantized Low-Rank Adaptation
QLoRA enables fine-tuning of large language models on consumer GPUs by combining 4-bit quantization with LoRA adapters. A 65B model can be fine-tuned on a single 48GB GPU while matching 16-bit fine-tuning performance.
Prerequisites: This skill assumes familiarity with LoRA. See the
loraskill for LoRA fundamentals (LoraConfig, target_modules, training patterns).
Table of Contents
- Core Innovations
- BitsAndBytesConfig Deep Dive
- Memory Requirements
- Complete Training Example
- Inference and Merging
- Troubleshooting
- Best Practices
- References
Core Innovations
QLoRA introduces three techniques that reduce memory usage without sacrificing performance:
4-bit NormalFloat (NF4)
NF4 is an information-theoretically optimal quantization data type for normally distributed weights. Neural network weights are typically normally distributed, making NF4 more efficient than standard 4-bit floats.
Storage: 4-bit NF4 (quantized weights)
Compute: 16-bit BF16 (dequantized for forward/backward pass)
The key insight: weights are stored in 4-bit but dequantized to bf16 for computation. Only the frozen base model is quantized; LoRA adapters remain in full precision.
NF4 vs FP4:
| Quantization | Description | Use Case |
|---|---|---|
nf4 | Normalized Float 4-bit, optimal for normal distributions | Default, recommended |
fp4 | Standard 4-bit float | Legacy, rarely needed |
Double Quantization
Standard quantization requires storing scaling constants (typically fp32) for each quantization block. Double quantization quantizes these constants too:
First quantization: weights → 4-bit + fp32 scaling constants
Double quantization: scaling constants → 8-bit + fp32 second-level constants
This saves approximately 0.37 bits per parameter—significant for billion-parameter models:
- 7B model: ~325 MB savings
- 70B model: ~3.2 GB savings
Paged Optimizers
During training, gradient checkpointing can cause memory spikes when processing long sequences. Paged optimizers use NVIDIA unified memory to automatically transfer optimizer states between GPU and CPU:
Normal training: OOM on memory spike
Paged optimizers: GPU ↔ CPU transfer handles spikes gracefully
This is handled automatically by bitsandbytes when using 4-bit training.
BitsAndBytesConfig Deep Dive
All Parameters Explained
from transformers import BitsAndBytesConfig
import torch
bnb_config = BitsAndBytesConfig(
# Core 4-bit settings
load_in_4bit=True, # Enable 4-bit quantization
bnb_4bit_quant_type="nf4", # "nf4" (recommended) or "fp4"
# Double quantization
bnb_4bit_use_double_quant=True, # Quantize the quantization constants
# Compute precision
bnb_4bit_compute_dtype=torch.bfloat16, # Dequantize to this dtype for compute
# Optional: specific storage type (usually auto-detected)
bnb_4bit_quant_storage=torch.uint8, # Storage dtype for quantized weights
)
Compute Dtype Selection
| Dtype | Hardware | Notes |
|---|---|---|
torch.bfloat16 | Ampere+ (RTX 30xx, A100) | Recommended, faster |
torch.float16 | Older GPUs (V100, RTX 20xx) | Use if bf16 not supported |
torch.float32 | Any | Slower, only for debugging |
Check bf16 support:
import torch
print(torch.cuda.is_bf16_supported()) # True on Ampere+
Comparison: Quantization Options
# Recommended: NF4 + double quant + bf16
optimal_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.bfloat16,
)
# Fallback when bf16 is unsupported
fp16_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.float16, # use when bf16 is unsupported or slower
)
# 8-bit alternative (less compression, sometimes more stable)
eight_bit_config = BitsAndBytesConfig(
load_in_8bit=True,
)
Memory Requirements
| Model Size | Full Fine-tuning | LoRA (16-bit) | QLoRA (4-bit) |
|---|---|---|---|
| 7B | ~60 GB | ~16 GB | ~6 GB |
| 13B | ~104 GB | ~28 GB | ~10 GB |
| 34B | ~272 GB | ~75 GB | ~20 GB |
| 70B | ~560 GB | ~160 GB | ~48 GB |
Notes:
- QLoRA memory includes model + optimizer states + activations
- Actual usage varies with batch size, sequence length, and gradient checkpointing
- Add ~20% buffer for safe operation
GPU Recommendations
| GPU VRAM | Max Model Size (QLoRA) |
|---|---|
| 8 GB | 7B (tight) |
| 16 GB | 7-13B |
| 24 GB | 13-34B |
| 48 GB | 34-70B |
| 80 GB | 70B+ comfortably |
Complete Training Example
from transformers import (
AutoModelForCausalLM,
AutoTokenizer,
BitsAndBytesConfig,
TrainingArguments,
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset
import torch
# 1. Quantization config
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.bfloat16,
)
# 2. Load quantized model
model_name = "meta-llama/Llama-3.1-8B"
model = AutoModelForCausalLM.from_pretrained(
model_name,
quantization_config=bnb_config,
device_map="auto",
dtype="auto",
attn_implementation="flash_attention_2", # Optional: faster attention
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"
# 3. Prepare for k-bit training (critical step!)
model = prepare_model_for_kbit_training(model)
# 4. LoRA config (see lora skill for parameter details)
lora_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# 5. Dataset
dataset = load_dataset("tatsu-lab/alpaca", split="train[:1000]")
def format_example(example):
if example["input"]:
return {"text": f"### Instruction:\n{example['instruction']}\n\n### Input:\n{example['input']}\n\n### Response:\n{example['output']}"}
return {"text": f"### Instruction:\n{example['instruction']}\n\n### Response:\n{example['output']}"}
dataset = dataset.map(format_example)
# 6. Training
sft_config = SFTConfig(
output_dir="./qlora-output",
max_length=512,
dataset_text_field="text",
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
num_train_epochs=1,
learning_rate=2e-4,
bf16=True,
logging_steps=10,
save_steps=100,
gradient_checkpointing=True,
gradient_checkpointing_kwargs={"use_reentrant": False},
optim="paged_adamw_8bit", # Paged optimizer for memory efficiency
)
trainer = SFTTrainer(
model=model,
args=sft_config,
train_dataset=dataset,
processing_class=tokenizer,
)
trainer.train()
# 7. Save adapter
model.save_pretrained("./qlora-adapter")
tokenizer.save_pretrained("./qlora-adapter")
Inference and Merging
Inference with Quantized Model
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch
model_name = "meta-llama/Llama-3.1-8B"
# Load quantized base model
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
)
base_model = AutoModelForCausalLM.from_pretrained(
model_name,
quantization_config=bnb_config,
device_map="auto",
dtype="auto",
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
# Load adapter
model = PeftModel.from_pretrained(base_model, "./qlora-adapter")
model.eval()
# Generate
inputs = tokenizer("### Instruction:\nExplain quantum computing.\n\n### Response:\n", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Merging to Full Precision
To merge QLoRA adapters into a full-precision model (for deployment without bitsandbytes):
from transformers import AutoModelForCausalLM
from peft import PeftModel
import torch
# Load base model in full precision (on CPU to avoid OOM)
base_model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.1-8B",
dtype=torch.bfloat16,
device_map="cpu",
)
# Load adapter
model = PeftModel.from_pretrained(base_model, "./qlora-adapter")
# Merge and unload
merged_model = model.merge_and_unload()
# Save merged model
merged_model.save_pretrained("./merged-model")
Note: Merging requires enough RAM to hold the full-precision model. For 70B models, this means ~140GB RAM.
Troubleshooting
CUDA Version Issues
# Check CUDA version
nvcc --version
python -c "import torch; print(torch.version.cuda)"
# bitsandbytes requires CUDA 11.7+
# If version mismatch, reinstall:
pip uninstall bitsandbytes
pip install bitsandbytes --upgrade
"cannot find libcudart" or Missing Library Errors
# Find CUDA installation
find /usr -name "libcudart*" 2>/dev/null
# Set environment variable
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
# Or for conda:
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH
Slow Training
Common cause: compute dtype mismatch
# Check if model is using expected dtype
for name, param in model.named_parameters():
if param.requires_grad:
print(f"{name}: {param.dtype}")
break # All LoRA params should match
# Ensure bf16 is used in training args if BitsAndBytesConfig uses bf16
# Mismatch causes constant dtype conversions
Out of Memory
# 1. Enable gradient checkpointing
model.gradient_checkpointing_enable()
# 2. Reduce batch size, increase accumulation
per_device_train_batch_size = 1
gradient_accumulation_steps = 16
# 3. Use paged optimizer
optim = "paged_adamw_8bit"
# 4. Reduce sequence length
max_length = 256
# 5. Target fewer modules
target_modules = ["q_proj", "v_proj"] # Minimal set
Model Loads But Training Fails
# Ensure prepare_model_for_kbit_training is called
from peft import prepare_model_for_kbit_training
model = prepare_model_for_kbit_training(model) # Don't skip this!
# Enable input gradients if needed
model.enable_input_require_grads()
Best Practices
-
Always use
prepare_model_for_kbit_training: This enables gradient computation through the frozen quantized layers -
Match compute dtype with training precision: If
bnb_4bit_compute_dtype=torch.bfloat16, usebf16=Truein training args -
Use paged optimizers for large models:
optim="paged_adamw_8bit"or"paged_adamw_32bit"handles memory spikes -
Start with NF4 + double quantization: This is the recommended default; only change if debugging
-
Gradient checkpointing is essential: Always enable for QLoRA training to fit larger batch sizes
-
Test inference before long training runs: Load the model and generate a few tokens to catch configuration issues early
-
Monitor GPU memory: Use
nvidia-smiortorch.cuda.memory_summary()to track actual usage -
Consider 8-bit for unstable training: If 4-bit training shows instability, try
load_in_8bit=Trueas a middle ground -
Use current TRL config fields: Put
max_lengthanddataset_text_fieldinSFTConfig, and pass the tokenizer or processor asprocessing_class.
References
Understanding RLHF
Reinforcement Learning from Human Feedback (RLHF) is a technique for aligning language models with human preferences. Rather than relying solely on next-token prediction, RLHF uses human judgment to guide model behavior toward helpful, harmless, and honest outputs.
Table of Contents
- Core Concepts
- The RLHF Pipeline
- Preference Data
- Instruction Tuning
- Reward Modeling
- Policy Optimization
- Direct Alignment Algorithms
- Challenges
- Best Practices
- References
Core Concepts
Why RLHF?
Pretraining produces models that predict likely text, not necessarily good text. A model trained on internet data learns to complete text in ways that reflect its training distribution—including toxic, unhelpful, or dishonest patterns. RLHF addresses this gap by optimizing for human preferences rather than likelihood.
The core insight: humans can often recognize good outputs more easily than they can specify what makes an output good. RLHF exploits this by collecting human judgments and using them to shape model behavior.
The Alignment Problem
Language models face several alignment challenges:
- Helpfulness: Following instructions and providing useful information
- Harmlessness: Avoiding toxic, dangerous, or inappropriate outputs
- Honesty: Acknowledging uncertainty and avoiding fabrication
- Intent alignment: Understanding what users actually want, not just what they say
RLHF provides a framework for encoding these properties through preference data.
Key Components
- Preference data: Human judgments comparing model outputs
- Reward model: A learned function approximating human preferences
- Policy optimization: RL algorithms that maximize expected reward
- Regularization: Constraints preventing deviation from the base model
The RLHF Pipeline
The standard RLHF pipeline consists of three main stages:
Stage 1: Supervised Fine-Tuning (SFT)
Start with a pretrained language model and fine-tune it on high-quality demonstrations. This teaches the model the desired format and style of responses.
Input: Pretrained model + demonstration dataset Output: SFT model that can follow instructions
Stage 2: Reward Model Training
Train a model to predict human preferences between pairs of outputs. The reward model learns to score outputs in a way that correlates with human judgment.
Input: SFT model + preference dataset (chosen/rejected pairs) Output: Reward model that scores any output
Stage 3: Policy Optimization
Use reinforcement learning to optimize the SFT model against the reward model, while staying close to the original SFT distribution.
Input: SFT model + reward model Output: Final aligned model
Alternative: Direct Alignment
Direct alignment algorithms (DPO, IPO, KTO) skip the reward model entirely, optimizing directly from preference data. This simplifies the pipeline but trades off some flexibility.
Modern Post-Training Variants
Current post-training stacks often mix these stages rather than using a single linear pipeline:
| Method | Data Signal | Best Fit |
|---|---|---|
| SFT | Demonstrations | Format, style, instruction following |
| DPO/IPO/KTO/ORPO | Offline preferences or binary feedback | Simpler alignment without online rollouts |
| PPO/RLOO | Reward model scores on sampled responses | Reward-model RL with explicit KL control |
| GRPO | Grouped completions scored by reward functions/models | Reasoning and verifiable-task optimization |
Preference Data
Preference data encodes human judgment about model outputs. The most common format is pairwise comparisons.
Pairwise Preferences
Given a prompt, collect two or more model outputs and have humans indicate which is better:
Prompt: "Explain quantum entanglement"
Response A: [technical explanation]
Response B: [simpler explanation with analogy]
Human preference: B > A
This creates (prompt, chosen, rejected) tuples for training.
Collection Methods
Human annotation: Trained annotators compare outputs according to guidelines. Most reliable but expensive and slow.
AI feedback: Use a capable model to generate preferences. Faster and cheaper but may propagate biases. This is the basis for Constitutional AI (CAI) and RLAIF.
Implicit signals: User interactions like upvotes, regeneration requests, or conversation length. Noisy but abundant.
Data Quality Considerations
- Annotator agreement: Low agreement suggests ambiguous criteria or subjective preferences
- Distribution coverage: Preferences should cover the range of model behaviors
- Prompt diversity: Diverse prompts prevent overfitting to narrow scenarios
- Preference strength: Some comparisons are clear; others are nearly ties
Instruction Tuning
Instruction tuning (supervised fine-tuning on instruction-response pairs) serves as the foundation for RLHF.
Purpose
- Teaches the model to follow instructions rather than just complete text
- Establishes the format and style for responses
- Creates a starting point that already exhibits desired behaviors
- Provides the reference policy for KL regularization
Dataset Composition
Typical instruction tuning datasets include:
- Single-turn QA: Questions with direct answers
- Multi-turn dialogue: Conversational exchanges
- Task instructions: Specific tasks with examples
- Chain-of-thought: Reasoning traces for complex problems
Relationship to RLHF
The SFT model defines the "prior" that RLHF refines. A better SFT model means:
- The reward model has better starting outputs to compare
- Policy optimization has less work to do
- The KL penalty keeps the final model closer to this baseline
Reward Modeling
The reward model transforms pairwise preferences into a scalar signal for RL optimization.
The Bradley-Terry Model
Preferences are modeled using the Bradley-Terry framework:
P(A > B) = sigmoid(r(A) - r(B))
Where r(x) is the reward for output x. This assumes preferences depend only on the difference in rewards.
The loss function is:
L = -log(sigmoid(r(chosen) - r(rejected)))
This pushes the reward model to assign higher scores to chosen outputs.
Architecture
Reward models are typically:
- The SFT model with a scalar head instead of the language modeling head
- Trained on (prompt, chosen, rejected) tuples
- Output a single scalar reward for any (prompt, response) pair
Considerations
- Scaling: Larger reward models generally produce better signals
- Calibration: Absolute reward values are less important than rankings
- Generalization: The model must score outputs it hasn't seen during training
- Over-optimization: Policies can exploit reward model weaknesses
See reference/reward-modeling.md for detailed training procedures.
Policy Optimization
Policy optimization uses RL to maximize expected reward while staying close to the reference policy.
The RLHF Objective
maximize E[R(x, y)] - β * KL(π || π_ref)
Where:
- R(x, y) is the reward for response y to prompt x
- KL(π || π_ref) measures deviation from the reference policy
- β controls the strength of the regularization
PPO (Proximal Policy Optimization)
PPO is the classic reward-model RLHF algorithm:
- Sample responses from the current policy
- Score responses with the reward model
- Compute advantage estimates
- Update policy with clipped surrogate objective
The clipping prevents large policy updates that could destabilize training.
KL Regularization
The KL penalty serves multiple purposes:
- Prevents reward hacking: Stops the policy from finding adversarial inputs to the reward model
- Maintains capabilities: Keeps the model close to the pretrained distribution
- Stabilizes training: Limits how far the policy can move per update
Higher β means more conservative updates; lower β allows more aggressive optimization.
REINFORCE vs PPO
REINFORCE is simpler but has higher variance:
- Uses raw returns without value function baseline
- Single-sample gradient estimates
- Can work for simpler problems
PPO adds complexity but improves stability:
- Clipped surrogate objective
- Multiple epochs per batch
- Better sample efficiency
See reference/policy-optimization.md for algorithm details.
GRPO (Group Relative Policy Optimization)
GRPO samples multiple completions for each prompt, scores them with reward functions or reward models, and normalizes advantages within the group. It removes the separate value model used by PPO, which can reduce memory and implementation complexity for LLM post-training.
Use GRPO when:
- Rewards can be computed automatically, such as math correctness, unit tests, format checks, or model-graded rubrics
- You want online learning from the model's own sampled completions
- Memory pressure makes PPO's value model unattractive
Watch for:
- Reward scaling choices, which can bias learning toward easier or shorter prompts
- Completion length bias
- Reward functions that are brittle or easy to exploit
Direct Alignment Algorithms
Direct alignment methods optimize the RLHF objective without training a separate reward model.
DPO (Direct Preference Optimization)
DPO reparameterizes the RLHF objective to derive a closed-form loss:
L = -log sigmoid(β * (log π(y_w|x)/π_ref(y_w|x) - log π(y_l|x)/π_ref(y_l|x)))
Where y_w is the preferred response and y_l is the dispreferred response.
Advantages:
- No separate reward model training
- Simpler pipeline with fewer hyperparameters
- More stable than RL-based methods
Trade-offs:
- Less flexible than explicit reward models
- Cannot reuse reward model for other purposes
- May be more sensitive to data quality
IPO (Identity Preference Optimization)
IPO addresses potential overfitting in DPO by using a different loss formulation that doesn't assume the Bradley-Terry model perfectly describes preferences.
KTO (Kahneman-Tversky Optimization)
KTO works with binary feedback (good/bad) rather than pairwise comparisons, making data collection easier. It's based on prospect theory from behavioral economics.
When to Use Direct Alignment
Direct alignment is preferred when:
- Simplicity is important
- Computational resources are limited
- The reward model won't be reused
Reward-based RLHF is preferred when:
- You need the reward model for other purposes (filtering, ranking)
- You have a large preference dataset
- You want maximum flexibility in optimization
See reference/direct-alignment.md for detailed algorithm comparisons.
Challenges
Over-Optimization
As optimization proceeds, the policy may exploit weaknesses in the reward model rather than improving on the true objective. Symptoms include:
- Rising reward model scores but declining human evaluation
- Increasingly verbose or formulaic outputs
- Sycophantic behavior (agreeing with users regardless of correctness)
Mitigations:
- Stronger KL regularization
- Reward model ensembles
- Early stopping based on held-out evaluation
Reward Hacking
The policy finds inputs that score highly with the reward model but don't represent genuine improvement:
- Length exploitation (longer responses score higher)
- Style mimicry (copying patterns from high-reward examples)
- Adversarial outputs that confuse the reward model
Evaluation
Evaluating aligned models is difficult because:
- Human preferences are subjective and context-dependent
- Automated metrics don't capture alignment properties well
- A/B testing is expensive and slow
- Models may perform differently on evaluation vs deployment
Distribution Shift
The preference data comes from a specific distribution of prompts and responses. The deployed model will encounter different inputs, and the reward model may not generalize well.
Best Practices
- Start with a strong SFT model: RLHF refines behavior; it works best when the base model already exhibits desired patterns
- Invest in preference data quality: Garbage in, garbage out—clear guidelines and trained annotators matter
- Use KL regularization: Don't optimize reward too aggressively; the reward model is an imperfect proxy
- Monitor for reward hacking: Track human evaluations alongside reward model scores
- Consider direct alignment first: DPO/ORPO-style methods are simpler than online RL when offline preference data is enough
- Iterate on reward model: Improve the reward model as you discover its weaknesses
- Diverse prompts: Ensure preference data covers the distribution you care about
- Regularize appropriately: Higher β for safety-critical applications; lower β for capability-focused training
- Use GRPO for verifiable reasoning tasks: Prefer reward functions with objective checks before model-graded rubrics
References
Reference Files
reference/reward-modeling.md- Detailed reward model training proceduresreference/policy-optimization.md- PPO and policy gradient algorithms for RLHFreference/direct-alignment.md- DPO, IPO, KTO and other direct methods
External Resources
- RLHF Book by Nathan Lambert - Comprehensive textbook on RLHF
- Training language models to follow instructions with human feedback - InstructGPT paper
- Direct Preference Optimization - DPO paper
- TRL documentation - Current SFT, DPO, GRPO, reward modeling, RLOO, and PPO trainers
- DeepSeekMath - Introduces GRPO
Using Hugging Face Transformers
Transformers is the model-definition framework for state-of-the-art machine learning across text, vision, audio, and multimodal domains. It provides unified APIs for loading pretrained models, running inference, and fine-tuning.
Table of Contents
- Core Concepts
- Pipeline API
- Model Loading
- Inference Patterns
- Fine-tuning with Trainer
- Working with Modalities
- Memory and Performance
- Best Practices
- References
Core Concepts
The Three Core Classes
Every model in Transformers has three core components:
from transformers import AutoConfig, AutoModel, AutoTokenizer, AutoProcessor
# Configuration: hyperparameters and architecture settings
config = AutoConfig.from_pretrained("bert-base-uncased")
# Model: the neural network weights
model = AutoModel.from_pretrained("bert-base-uncased")
# Tokenizer: converts text inputs to tensors
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
# Processor: unified preprocessing for vision, audio, and multimodal models
processor = AutoProcessor.from_pretrained("openai/whisper-large-v3")
The from_pretrained Pattern
All loading uses from_pretrained() which handles downloading, caching, and device placement:
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_name = "meta-llama/Llama-3.2-1B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
model_name,
dtype=torch.bfloat16,
device_map="auto", # Automatic device placement
)
Transformers v5 examples use dtype. On Transformers v4, the equivalent argument is torch_dtype.
Auto Classes
Use task-specific Auto classes for the correct model head:
from transformers import (
AutoModelForCausalLM, # Text generation (GPT, Llama)
AutoModelForSeq2SeqLM, # Encoder-decoder (T5, BART)
AutoModelForSequenceClassification, # Classification
AutoModelForTokenClassification, # NER, POS tagging
AutoModelForQuestionAnswering, # Extractive QA
AutoModelForMaskedLM, # BERT-style masked LM
AutoModelForImageClassification, # Vision models
AutoModelForSpeechSeq2Seq, # Speech recognition
)
Pipeline API
The pipeline() function provides high-level inference with minimal code:
Text Tasks
from transformers import pipeline
# Text generation
generator = pipeline("text-generation", model="Qwen/Qwen2.5-1.5B")
output = generator("The secret to success is", max_new_tokens=50)
# Text classification
classifier = pipeline("sentiment-analysis")
result = classifier("I love this product!")
# [{'label': 'POSITIVE', 'score': 0.9998}]
# Named entity recognition
ner = pipeline("ner", aggregation_strategy="simple")
entities = ner("Hugging Face is based in New York City.")
# Question answering
qa = pipeline("question-answering")
answer = qa(question="What is the capital?", context="Paris is the capital of France.")
# Summarization
summarizer = pipeline("summarization", model="facebook/bart-large-cnn")
summary = summarizer(long_text, max_length=130, min_length=30)
# Translation
translator = pipeline("translation_en_to_fr", model="Helsinki-NLP/opus-mt-en-fr")
result = translator("Hello, how are you?")
Chat/Conversational
from transformers import pipeline
import torch
pipe = pipeline(
"text-generation",
model="meta-llama/Llama-3.2-3B-Instruct",
dtype=torch.bfloat16,
device_map="auto",
)
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain quantum computing in simple terms."},
]
response = pipe(messages, max_new_tokens=256)
print(response[0]["generated_text"][-1]["content"])
Vision Tasks
classifier = pipeline("image-classification", model="google/vit-base-patch16-224")
detector = pipeline("object-detection", model="facebook/detr-resnet-50")
Audio Tasks
transcriber = pipeline("automatic-speech-recognition", model="openai/whisper-large-v3")
text = transcriber("path/to/audio.mp3")
Multimodal Tasks
vqa = pipeline("visual-question-answering", model="Salesforce/blip-vqa-base")
captioner = pipeline("image-to-text", model="Salesforce/blip-image-captioning-base")
Model Loading
Device Placement
from transformers import AutoModelForCausalLM
import torch
# Automatic placement across available devices
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.2-3B",
device_map="auto",
dtype=torch.bfloat16,
)
# Specific device
model = AutoModelForCausalLM.from_pretrained(
"gpt2",
device_map="cuda:0",
)
# Custom device map for model parallelism
device_map = {
"model.embed_tokens": 0,
"model.layers.0": 0,
"model.layers.1": 1,
"model.norm": 1,
"lm_head": 1,
}
model = AutoModelForCausalLM.from_pretrained(model_name, device_map=device_map)
Loading from Local Path
# Save model locally
model.save_pretrained("./my_model")
tokenizer.save_pretrained("./my_model")
# Load from local path
model = AutoModelForCausalLM.from_pretrained("./my_model")
tokenizer = AutoTokenizer.from_pretrained("./my_model")
Trust Remote Code
Some models require executing custom code from the Hub:
model = AutoModelForCausalLM.from_pretrained(
"microsoft/phi-2",
trust_remote_code=True, # Required for custom architectures
)
Prefer models with safetensors weights when available. Safetensors avoids pickle execution risks and typically loads faster than legacy .bin checkpoints.
Inference Patterns
Text Generation
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_name = "Qwen/Qwen2.5-3B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
model_name,
dtype=torch.bfloat16,
device_map="auto",
)
# Basic generation
inputs = tokenizer("Once upon a time", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
text = tokenizer.decode(outputs[0], skip_special_tokens=True)
# With generation config
outputs = model.generate(
**inputs,
max_new_tokens=100,
do_sample=True,
temperature=0.7,
top_p=0.9,
top_k=50,
repetition_penalty=1.1,
)
Chat Templates
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"},
]
# Apply chat template
input_text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
)
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
Getting Embeddings
from transformers import AutoModel, AutoTokenizer
import torch
model = AutoModel.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")
tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")
def get_embeddings(texts: list[str]) -> torch.Tensor:
inputs = tokenizer(texts, padding=True, truncation=True, return_tensors="pt")
with torch.no_grad():
outputs = model(**inputs)
# Mean pooling
attention_mask = inputs["attention_mask"]
embeddings = outputs.last_hidden_state
mask_expanded = attention_mask.unsqueeze(-1).expand(embeddings.size()).float()
sum_embeddings = (embeddings * mask_expanded).sum(1)
sum_mask = mask_expanded.sum(1).clamp(min=1e-9)
return sum_embeddings / sum_mask
embeddings = get_embeddings(["Hello world", "How are you?"])
Classification
from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch
model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english")
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english")
inputs = tokenizer("I love this movie!", return_tensors="pt")
with torch.no_grad():
outputs = model(**inputs)
predictions = torch.softmax(outputs.logits, dim=-1)
labels = model.config.id2label
for idx, prob in enumerate(predictions[0]):
print(f"{labels[idx]}: {prob:.4f}")
Fine-tuning with Trainer
Basic Fine-tuning
from transformers import (
AutoModelForSequenceClassification,
AutoTokenizer,
Trainer,
TrainingArguments,
)
from datasets import load_dataset
# Load data and model
dataset = load_dataset("imdb")
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
model = AutoModelForSequenceClassification.from_pretrained(
"distilbert-base-uncased",
num_labels=2,
)
# Tokenize dataset
def tokenize(examples):
return tokenizer(examples["text"], padding="max_length", truncation=True)
tokenized = dataset.map(tokenize, batched=True)
# Training arguments
training_args = TrainingArguments(
output_dir="./results",
eval_strategy="epoch",
learning_rate=2e-5,
per_device_train_batch_size=16,
per_device_eval_batch_size=16,
num_train_epochs=3,
weight_decay=0.01,
logging_steps=100,
save_strategy="epoch",
load_best_model_at_end=True,
)
# Train
trainer = Trainer(
model=model,
args=training_args,
train_dataset=tokenized["train"],
eval_dataset=tokenized["test"],
)
trainer.train()
Pushing to Hub
# Login first: huggingface-cli login
# Push model and tokenizer
model.push_to_hub("my-username/my-fine-tuned-model")
tokenizer.push_to_hub("my-username/my-fine-tuned-model")
# Or use trainer
trainer.push_to_hub()
See reference/fine-tuning.md for advanced patterns including LoRA, custom data collators, and evaluation metrics.
Working with Modalities
Use AutoProcessor or modality-specific processors for non-text models. Processors handle images, audio, video, and multimodal chat formatting before tensors are sent to the model.
| Modality | Processor | Model Class |
|---|---|---|
| Vision | AutoImageProcessor | AutoModelForImageClassification, AutoModelForObjectDetection |
| Audio | AutoProcessor | AutoModelForSpeechSeq2Seq, AutoModelForAudioClassification |
| Vision-language | AutoProcessor | AutoModelForVision2Seq, task-specific VLM classes |
Memory and Performance
Quantization
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch
# 4-bit quantization
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.2-3B",
quantization_config=bnb_config,
device_map="auto",
dtype="auto",
)
# 8-bit quantization
bnb_config = BitsAndBytesConfig(load_in_8bit=True)
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.2-3B",
quantization_config=bnb_config,
device_map="auto",
dtype="auto",
)
Flash Attention
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.2-3B",
dtype=torch.bfloat16,
attn_implementation="flash_attention_2", # Requires flash-attn package
device_map="auto",
)
torch.compile
model = AutoModelForCausalLM.from_pretrained(model_name, dtype=torch.bfloat16)
model = torch.compile(model, mode="reduce-overhead")
Batched Inference
texts = ["First prompt", "Second prompt", "Third prompt"]
inputs = tokenizer(texts, return_tensors="pt", padding=True).to(model.device)
with torch.no_grad():
outputs = model.generate(**inputs, max_new_tokens=50)
decoded = tokenizer.batch_decode(outputs, skip_special_tokens=True)
Best Practices
- Use bfloat16 over float16: Better numerical stability on modern GPUs
- Use
dtype="auto"when unsure: It follows model metadata or checkpoint weight dtype without wasting memory - Set pad token for generation:
tokenizer.pad_token = tokenizer.eos_token - Use device_map="auto": Let Accelerate handle device placement
- Prefer safetensors checkpoints: Avoid pickle-based loading when possible
- Enable Flash Attention: Significant speedup for long sequences
- Batch when possible: Amortize fixed costs across multiple inputs
- Use pipeline for quick prototyping: Switch to manual control for production
- Cache models locally: Set
HF_HOMEenvironment variable for model cache location - Check model license: Verify usage rights before deployment
References
See reference/ for detailed documentation:
fine-tuning.md- Advanced fine-tuning patterns with LoRA, PEFT, and custom training
External documentation:
Sponsors
Details
- Marketplace
- itsmostafa/llm-engineering-skills
- Category
- Community