itsmostafa avatar

llm-engineering-skills

Collection of LLM engineering skills for PyTorch, Transformers, LoRA, and MLX

GitHub stars
25

Install

/plugin install llm-engineering-skills@llm-engineering-skills

Run it inside Claude Code after adding the marketplace with /plugin marketplace add itsmostafa/llm-engineering-skills.

Contents

Discovered from source

Skills 9

Building Agents

Agents are systems where LLMs dynamically direct their own processes and tool usage. This skill covers when to use agents vs workflows, common architectural patterns, and practical implementation guidance.

Table of Contents

Agents vs Workflows

AspectWorkflowsAgents
Control flowPredefined code pathsLLM determines next step
PredictabilityHigh - deterministic stepsLower - dynamic decisions
ComplexitySimpler to debug and testMore complex, harder to predict
Best forWell-defined, repeatable tasksOpen-ended, adaptive problems

Key principle: Start with the simplest solution. Use workflows when the task is predictable; use agents when flexibility is required.

Workflow Patterns

1. Prompt Chaining

Decompose tasks into sequential LLM calls, where each step's output feeds the next.

async def prompt_chain(input_text):
    # Step 1: Extract key information
    extracted = await llm.generate(
        "Extract the main entities and relationships from: " + input_text
    )

    # Step 2: Analyze
    analysis = await llm.generate(
        "Analyze these entities for patterns: " + extracted
    )

    # Step 3: Generate output
    return await llm.generate(
        "Based on this analysis, provide recommendations: " + analysis
    )

Use when: Tasks naturally decompose into fixed sequential steps.

2. Routing

Classify inputs and direct them to specialized handlers.

async def route_request(user_input):
    # Classify the input
    category = await llm.generate(
        f"Classify this request into one of: [billing, technical, general]\n{user_input}"
    )

    handlers = {
        "billing": handle_billing,
        "technical": handle_technical,
        "general": handle_general,
    }

    return await handlers[category.strip()](user_input)

Use when: Different input types need fundamentally different processing.

3. Parallelization

Run multiple LLM calls concurrently for independent subtasks.

import asyncio

async def parallel_analysis(document):
    # Run independent analyses in parallel
    results = await asyncio.gather(
        llm.generate(f"Summarize: {document}"),
        llm.generate(f"Extract key facts: {document}"),
        llm.generate(f"Identify sentiment: {document}"),
    )

    summary, facts, sentiment = results
    return {"summary": summary, "facts": facts, "sentiment": sentiment}

Variants:

  • Sectioning: Break task into parallel subtasks
  • Voting: Run same prompt multiple times, aggregate results

4. Orchestrator-Workers

Central LLM decomposes tasks and delegates to worker LLMs.

class Orchestrator:
    async def run(self, task):
        # Break down the task
        subtasks = await self.plan(task)

        # Delegate to workers
        results = []
        for subtask in subtasks:
            worker_result = await self.delegate(subtask)
            results.append(worker_result)

        # Synthesize results
        return await self.synthesize(results)

    async def plan(self, task):
        response = await llm.generate(
            f"Break this task into subtasks:\n{task}\n\nReturn as JSON array."
        )
        return json.loads(response)

    async def delegate(self, subtask):
        return await llm.generate(f"Complete this subtask:\n{subtask}")

    async def synthesize(self, results):
        return await llm.generate(
            f"Combine these results into a coherent response:\n{results}"
        )

Use when: Tasks require dynamic decomposition that can't be predetermined.

5. Evaluator-Optimizer

One LLM generates, another evaluates and requests improvements.

async def generate_with_feedback(task, max_iterations=3):
    response = await llm.generate(f"Complete this task:\n{task}")

    for _ in range(max_iterations):
        evaluation = await llm.generate(
            f"Evaluate this response for quality and correctness:\n{response}\n"
            "If improvements needed, specify them. Otherwise respond 'APPROVED'."
        )

        if "APPROVED" in evaluation:
            return response

        response = await llm.generate(
            f"Improve this response based on feedback:\n"
            f"Original: {response}\nFeedback: {evaluation}"
        )

    return response

Use when: Output quality is critical and can be objectively evaluated.

Agent Architectures

Autonomous Agent Loop

Agents operate in a loop: observe, think, act, repeat.

class Agent:
    def __init__(self, tools: list, system_prompt: str):
        self.tools = {t.name: t for t in tools}
        self.system_prompt = system_prompt

    async def run(self, task: str, max_steps: int = 10):
        messages = [
            {"role": "system", "content": self.system_prompt},
            {"role": "user", "content": task},
        ]

        for step in range(max_steps):
            response = await llm.generate(messages, tools=self.tools)
            messages.append({"role": "assistant", "content": response})

            if response.tool_calls:
                for call in response.tool_calls:
                    result = await self.execute_tool(call)
                    messages.append({
                        "role": "tool",
                        "tool_call_id": call.id,
                        "content": result
                    })
            else:
                # No tool calls - agent is done
                return response.content

        return "Max steps reached"

    async def execute_tool(self, call):
        tool = self.tools[call.name]
        return await tool.execute(**call.arguments)

Human-in-the-Loop

Pause for human approval at critical checkpoints.

class HumanInLoopAgent(Agent):
    def __init__(self, tools, system_prompt, approval_required: list):
        super().__init__(tools, system_prompt)
        self.approval_required = set(approval_required)

    async def execute_tool(self, call):
        if call.name in self.approval_required:
            approved = await self.request_approval(call)
            if not approved:
                return "Action cancelled by user"

        return await super().execute_tool(call)

    async def request_approval(self, call):
        print(f"Agent wants to execute: {call.name}({call.arguments})")
        response = input("Approve? (y/n): ")
        return response.lower() == "y"

ReAct Pattern

ReAct (Reasoning and Acting) alternates between thinking and taking actions.

REACT_PROMPT = """Answer the question using the available tools.

For each step:
1. Thought: Reason about what to do next
2. Action: Choose a tool and inputs
3. Observation: See the result
4. Repeat until you have the answer

Available tools: {tools}

Question: {question}
"""

async def react_agent(question, tools):
    prompt = REACT_PROMPT.format(
        tools=format_tools(tools),
        question=question
    )

    messages = [{"role": "user", "content": prompt}]

    while True:
        response = await llm.generate(messages)
        messages.append({"role": "assistant", "content": response})

        if "Final Answer:" in response:
            return extract_final_answer(response)

        action = parse_action(response)
        if action:
            observation = await execute_tool(action, tools)
            messages.append({
                "role": "user",
                "content": f"Observation: {observation}"
            })

Advantages:

  • Explicit reasoning traces aid debugging
  • More interpretable decision-making
  • Better handling of complex multi-step tasks

Tool Design

Principles

  1. Self-contained: Tools return complete, usable information
  2. Scoped: Each tool does one thing well
  3. Descriptive: Clear names and descriptions guide the LLM
  4. Error-robust: Return informative errors, not exceptions

Tool Definition Pattern

class Tool:
    def __init__(self, name: str, description: str, parameters: dict, fn):
        self.name = name
        self.description = description
        self.parameters = parameters
        self.fn = fn

    async def execute(self, **kwargs):
        try:
            return await self.fn(**kwargs)
        except Exception as e:
            return f"Error: {str(e)}"

# Example tool
search_tool = Tool(
    name="search_database",
    description="Search the database for records matching a query. "
                "Returns up to 10 matching records with their IDs and summaries.",
    parameters={
        "query": {"type": "string", "description": "Search query"},
        "limit": {"type": "integer", "description": "Max results (default 10)"},
    },
    fn=search_database
)

Tool Interface Guidelines

  • Prefer text inputs/outputs over complex structured data
  • Include usage examples in descriptions for ambiguous tools
  • Return truncated results when output could be large
  • Provide clear feedback on what the tool did

External Context Protocols

Model Context Protocol (MCP) standardizes how agents discover tools, prompts, and resources from external systems. Prefer a protocol boundary when multiple agents, apps, or runtimes need the same integration.

Use MCP-style boundaries when:

  • The integration exposes reusable tools or data resources
  • You need consistent auth and permission handling outside the agent loop
  • Tool metadata should be discoverable rather than hard-coded into prompts

Keep tool schemas tight:

  • Use narrow parameters with explicit enums where possible
  • Return structured summaries before raw payloads
  • Include stable resource IDs so the agent can fetch details on demand
  • Keep destructive actions behind explicit approval or dry-run modes

Best Practices

  1. Start simple: Begin with the simplest architecture that could work. Add complexity only when it demonstrably improves outcomes.

  2. Maintain transparency: Ensure the agent's planning steps are visible. This aids debugging and builds user trust.

  3. Design for failure: Agents will make mistakes. Include guardrails, retries, and graceful degradation.

  4. Test extensively: Use sandboxed environments. Test edge cases and failure modes, not just happy paths.

  5. Limit tool proliferation: More tools means more confusion. Keep the tool set focused and well-documented.

  6. Implement checkpoints: For long-running tasks, save state periodically to enable recovery.

  7. Set resource limits: Cap iterations, token usage, and tool calls to prevent runaway agents.

  8. Log everything: Record all LLM calls, tool executions, and decisions for debugging and improvement.

  9. Handle ambiguity: When uncertain, have the agent ask for clarification rather than guessing.

  10. Measure outcomes: Track task completion rates, accuracy, and efficiency to guide improvements.

  11. Separate integration boundaries: Use MCP or equivalent tool/resource servers when integrations need auth, shared state, or reuse across agents.

References

Context Engineering

Context engineering is the discipline of curating and maintaining the optimal set of tokens during LLM inference. Unlike prompt engineering (crafting individual prompts), context engineering focuses on what information enters the context window and when.

Table of Contents

Core Principles

Context as a Finite Resource

LLMs have limited "attention budgets." As context length increases, models experience context rot—decreased ability to accurately recall information. The goal is finding the smallest possible set of high-signal tokens that maximize desired outcomes.

Effective Context = Relevant Information / Total Tokens

Key insight: More context isn't better. The right context is better.

The Context Pollution Problem

Every token added to context has costs:

  • Increased latency and compute
  • Diluted attention to important information
  • Higher risk of hallucination from conflicting data
  • Reduced model performance on retrieval tasks

Context Management Strategies

1. Context Trimming

Drop older conversation turns, keeping only the last N turns.

AspectDetails
MechanismSliding window over conversation history
ProsDeterministic, zero latency, preserves recent context verbatim
ConsAbrupt loss of long-range context, "amnesia" effect
Best forIndependent tasks, short interactions, predictable workflows
def trim_context(messages: list, keep_last_n: int = 10) -> list:
    """Keep system message + last N turns."""
    system_msgs = [m for m in messages if m["role"] == "system"]
    other_msgs = [m for m in messages if m["role"] != "system"]
    return system_msgs + other_msgs[-keep_last_n:]

2. Context Summarization

Compress prior messages into structured summaries.

AspectDetails
MechanismLLM generates summary of older context
ProsRetains long-range memory, smoother UX, scalable
ConsSummarization bias risk, added latency, potential compounding errors
Best forComplex multi-step tasks, long-horizon interactions
SUMMARIZATION_PROMPT = """Summarize the conversation so far, preserving:
1. Key decisions made
2. Important context established
3. Current task state and goals
4. Any constraints or preferences expressed

Be concise but complete. Output as structured markdown."""

async def summarize_context(messages: list, model) -> str:
    """Generate a summary of conversation history."""
    conversation_text = format_messages_for_summary(messages)
    response = await model.generate(
        system=SUMMARIZATION_PROMPT,
        user=conversation_text
    )
    return response.content

3. Hybrid Approach

Combine trimming and summarization for optimal balance.

class HybridContextManager:
    def __init__(
        self,
        keep_recent: int = 5,      # Recent turns to keep verbatim
        summary_threshold: int = 20, # When to trigger summarization
    ):
        self.keep_recent = keep_recent
        self.summary_threshold = summary_threshold
        self.running_summary = ""

    def process(self, messages: list) -> list:
        if len(messages) < self.summary_threshold:
            return messages

        # Summarize older messages
        old_messages = messages[:-self.keep_recent]
        self.running_summary = summarize(old_messages, self.running_summary)

        # Return summary + recent messages
        return [
            {"role": "system", "content": f"Previous context:\n{self.running_summary}"},
            *messages[-self.keep_recent:]
        ]

4. Session Memory

Persist reusable facts, preferences, and task state outside the context window. Load only the relevant slice for the current turn.

AspectDetails
MechanismExternal store keyed by user, session, task, or resource
ProsRecovers long-range context without carrying all history
ConsRequires retrieval, freshness, and deletion policies
Best forAgents, project work, personalization, long-running workflows

Separate durable memory from ephemeral scratchpads. Durable memory should contain stable facts and explicit decisions, not every intermediate thought.

System Prompt Design

Principles for Context-Efficient Prompts

  1. Clear and direct language: Avoid ambiguity that requires clarification turns
  2. Structured sections: Organize by purpose (role, capabilities, constraints)
  3. Minimal yet comprehensive: Include only what affects behavior
  4. Self-contained instructions: Reduce need for context retrieval

Example Structure

# Role
You are [specific role] that [primary function].

# Capabilities
- [Capability 1 with scope]
- [Capability 2 with scope]

# Constraints
- [Hard constraint]
- [Preference]

# Output Format
[Specific format requirements]

Tool Design for Context Efficiency

Just-in-Time Context Loading

Instead of front-loading all possible context, load information dynamically as needed.

# Anti-pattern: Loading everything upfront
context = load_all_user_data()  # Large, mostly unused
context += load_all_documents()  # Even larger

# Better: Just-in-time retrieval
tools = [
    Tool(
        name="get_user_preference",
        description="Get specific user preference by key",
        # Only fetches what's needed when asked
    ),
    Tool(
        name="search_documents",
        description="Search documents by query",
        # Returns relevant subset
    ),
]

Tool Design Principles

  1. Self-contained: Each tool returns complete, usable information
  2. Scoped: Tools do one thing well
  3. Descriptive: Names and descriptions guide LLM toward correct usage
  4. Error-robust: Return informative errors that don't pollute context
# Well-designed tool
def search_codebase(query: str, max_results: int = 5) -> str:
    """Search codebase for relevant code snippets.

    Args:
        query: Natural language description of what to find
        max_results: Maximum snippets to return (default 5)

    Returns:
        Formatted code snippets with file paths and line numbers,
        or 'No results found' if nothing matches.
    """
    results = perform_search(query, limit=max_results)
    if not results:
        return "No results found for query."
    return format_results(results)  # Concise, structured output

Long-Horizon Task Patterns

Pattern 1: Compaction

Periodically compress conversation history to reclaim context space.

async def compaction_loop(agent, messages, task):
    while not task.complete:
        # Process next step
        response = await agent.run(messages)
        messages.append(response)

        # Compact when approaching limit
        if estimate_tokens(messages) > TOKEN_LIMIT * 0.8:
            summary = await summarize_context(messages[:-3])
            messages = [
                {"role": "system", "content": agent.system_prompt},
                {"role": "assistant", "content": f"Summary of progress:\n{summary}"},
                *messages[-3:]  # Keep recent context
            ]

    return messages

Pattern 2: Structured Note-Taking

Agent maintains external notes, retrieving as needed.

class NoteTakingAgent:
    def __init__(self):
        self.notes = {}  # Key-value store outside context

    async def run(self, messages):
        tools = [
            Tool("save_note", self.save_note, "Save information for later"),
            Tool("get_note", self.get_note, "Retrieve saved information"),
            Tool("list_notes", self.list_notes, "List all saved note keys"),
        ]
        return await self.agent.run(messages, tools=tools)

    def save_note(self, key: str, content: str) -> str:
        self.notes[key] = content
        return f"Saved note: {key}"

    def get_note(self, key: str) -> str:
        return self.notes.get(key, f"No note found for key: {key}")

Pattern 3: Sub-Agent Architecture

Delegate focused tasks to specialized agents with clean context.

class OrchestratorAgent:
    def __init__(self):
        self.sub_agents = {
            "researcher": ResearchAgent(),
            "coder": CodingAgent(),
            "reviewer": ReviewAgent(),
        }

    async def delegate(self, task: str, agent_type: str) -> str:
        """Delegate to sub-agent, receive condensed summary."""
        agent = self.sub_agents[agent_type]

        # Sub-agent works with fresh context
        result = await agent.run(task)

        # Return only essential findings to main context
        return result.summary  # Not the full conversation

Benefits:

  • Each sub-agent has focused, clean context
  • Main agent receives condensed results
  • Parallelization opportunities
  • Failure isolation

Implementation Patterns

Session Memory Manager

class SessionMemory:
    def __init__(
        self,
        keep_last_n_turns: int = 5,
        context_limit: int = 100_000,  # tokens
        summarizer = None,
    ):
        self.keep_last_n_turns = keep_last_n_turns
        self.context_limit = context_limit
        self.summarizer = summarizer
        self.messages = []
        self.summary = ""

    async def add_message(self, message: dict):
        self.messages.append(message)
        await self._maybe_compact()

    async def _maybe_compact(self):
        current_tokens = estimate_tokens(self.messages)

        if current_tokens > self.context_limit * 0.8:
            # Summarize all but recent messages
            old_messages = self.messages[:-self.keep_last_n_turns]
            new_summary = await self.summarizer.summarize(
                old_messages,
                previous_summary=self.summary
            )
            self.summary = new_summary
            self.messages = self.messages[-self.keep_last_n_turns:]

    def get_context(self) -> list:
        context = []
        if self.summary:
            context.append({
                "role": "system",
                "content": f"Conversation summary:\n{self.summary}"
            })
        context.extend(self.messages)
        return context

Token Estimation

def estimate_tokens(messages: list) -> int:
    """Rough token estimation (4 chars ≈ 1 token for English)."""
    total_chars = sum(
        len(m.get("content", ""))
        for m in messages
    )
    return total_chars // 4

def estimate_tokens_accurate(messages: list, model: str) -> int:
    """Accurate token count using tiktoken."""
    import tiktoken
    encoding = tiktoken.encoding_for_model(model)
    return sum(
        len(encoding.encode(m.get("content", "")))
        for m in messages
    )

Best Practices

  1. Treat context as precious: Every token has a cost. Include only information that improves task performance.

  2. Use progressive disclosure: Start minimal, expand context only when needed via tools.

  3. Design for recoverability: Agents should be able to reconstruct critical context from external sources.

  4. Monitor context health: Track token usage, retrieval accuracy, and task completion rates.

  5. Prefer structured over raw data: JSON, markdown tables, and clear formatting improve information density.

  6. Implement graceful degradation: When context limits approach, prioritize recent and high-signal information.

  7. Test with long conversations: Validate agent behavior after many turns, not just initial interactions.

  8. Separate concerns: Use different context regions for system instructions, user history, and tool outputs.

  9. Version your summaries: When compacting, maintain enough structure to debug summarization issues.

  10. Measure and iterate: Context engineering is empirical—test what information actually improves outcomes.

  11. Separate memory classes: Keep instructions, recent messages, durable memory, retrieved documents, and tool results in distinct sections so conflicts are easier to detect.

References

Using LoRA for Fine-tuning

LoRA (Low-Rank Adaptation) enables efficient fine-tuning by freezing pretrained weights and injecting small trainable matrices into transformer layers. This reduces trainable parameters to ~0.1% of the original model while maintaining performance.

Table of Contents

Core Concepts

How LoRA Works

Instead of updating all weights during fine-tuning, LoRA decomposes weight updates into low-rank matrices:

W' = W + BA

Where:

  • W is the frozen pretrained weight matrix (d × k)
  • B is a trainable matrix (d × r)
  • A is a trainable matrix (r × k)
  • r is the rank, much smaller than d and k

The key insight: weight updates during fine-tuning have low intrinsic rank, so we can represent them efficiently with smaller matrices.

Why Use LoRA

AspectFull Fine-tuningLoRA
Trainable params100%~0.1-1%
Memory usageHighLow
Adapter sizeFull model~3-100 MB
Training speedSlowerFaster
Multiple tasksSeparate modelsSwap adapters

Basic Setup

Installation

pip install peft transformers accelerate

Minimal Example

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model, TaskType
import torch

# Load base model
model_name = "meta-llama/Llama-3.2-1B"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token

# Configure LoRA
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM,
)

# Apply LoRA
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 3,407,872 || all params: 1,238,300,672 || trainable%: 0.28%

Configuration Parameters

LoraConfig Options

from peft import LoraConfig, TaskType

config = LoraConfig(
    # Core parameters
    r=16,                          # Rank of update matrices
    lora_alpha=32,                 # Scaling factor (alpha/r applied to updates)
    target_modules=["q_proj", "v_proj"],  # Layers to adapt

    # Regularization
    lora_dropout=0.05,             # Dropout on LoRA layers
    bias="none",                   # "none", "all", or "lora_only"

    # Task configuration
    task_type=TaskType.CAUSAL_LM,  # CAUSAL_LM, SEQ_CLS, SEQ_2_SEQ_LM, TOKEN_CLS

    # Advanced
    modules_to_save=None,          # Additional modules to train (e.g., ["lm_head"])
    layers_to_transform=None,      # Specific layer indices to adapt
    rank_pattern=None,             # Per-module rank overrides
    alpha_pattern=None,            # Per-module alpha overrides
    trainable_token_indices=None,  # Train only selected new token embeddings
    target_parameters=None,        # Target nn.Parameter weights in some MoE layers
    use_rslora=False,              # Rank-stabilized LoRA scaling
    use_dora=False,                # Weight-Decomposed LoRA
)

Target Modules by Architecture

# Llama, Mistral, Qwen
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]

# GPT-2, GPT-J
target_modules = ["c_attn", "c_proj", "c_fc"]

# BERT, RoBERTa
target_modules = ["query", "key", "value", "dense"]

# Falcon
target_modules = ["query_key_value", "dense", "dense_h_to_4h", "dense_4h_to_h"]

# Phi
target_modules = ["q_proj", "k_proj", "v_proj", "dense", "fc1", "fc2"]

# MoE expert parameters stored as nn.Parameter tensors
target_parameters = ["feed_forward.experts.gate_up_proj", "feed_forward.experts.down_proj"]

For MoE models where expert weights are not nn.Linear modules, use target_parameters rather than target_modules. Merge adapters before latency-sensitive inference because materializing expert LoRA updates can add overhead.

Finding Target Modules

# Print all linear layer names
from peft.utils import get_peft_model_state_dict

def find_target_modules(model):
    linear_modules = set()
    for name, module in model.named_modules():
        if isinstance(module, torch.nn.Linear):
            # Get the last part of the name (e.g., "q_proj" from "model.layers.0.self_attn.q_proj")
            layer_name = name.split(".")[-1]
            linear_modules.add(layer_name)
    return list(linear_modules)

print(find_target_modules(model))

QLoRA (Quantized LoRA)

QLoRA combines 4-bit quantization with LoRA, enabling fine-tuning of large models on consumer GPUs.

Setup

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
import torch

# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",           # Normalized float 4-bit
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,       # Nested quantization
)

# Load quantized model
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B",
    quantization_config=bnb_config,
    device_map="auto",
)

# Prepare for k-bit training
model = prepare_model_for_kbit_training(model)

# Apply LoRA
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM,
)

model = get_peft_model(model, lora_config)

Memory Requirements

Model SizeFull FT (16-bit)LoRA (16-bit)QLoRA (4-bit)
7B~60 GB~16 GB~6 GB
13B~104 GB~28 GB~10 GB
70B~560 GB~160 GB~48 GB

Training Patterns

With Hugging Face Trainer

from transformers import TrainingArguments, Trainer, DataCollatorForLanguageModeling
from datasets import load_dataset

# Prepare dataset
dataset = load_dataset("tatsu-lab/alpaca", split="train")

def format_prompt(example):
    if example["input"]:
        text = f"### Instruction:\n{example['instruction']}\n\n### Input:\n{example['input']}\n\n### Response:\n{example['output']}"
    else:
        text = f"### Instruction:\n{example['instruction']}\n\n### Response:\n{example['output']}"
    return {"text": text}

dataset = dataset.map(format_prompt)

def tokenize(examples):
    return tokenizer(
        examples["text"],
        truncation=True,
        max_length=512,
        padding=False,
    )

tokenized = dataset.map(tokenize, batched=True, remove_columns=dataset.column_names)

# Training arguments (note higher learning rate)
training_args = TrainingArguments(
    output_dir="./lora-output",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    num_train_epochs=1,
    learning_rate=2e-4,              # Higher than full fine-tuning
    bf16=True,
    logging_steps=10,
    save_steps=500,
    warmup_ratio=0.03,
    gradient_checkpointing=True,
    optim="adamw_torch_fused",
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized,
    data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False),
)

trainer.train()

With SFTTrainer (TRL)

from trl import SFTTrainer, SFTConfig

sft_config = SFTConfig(
    output_dir="./sft-lora",
    max_length=1024,
    dataset_text_field="text",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    num_train_epochs=1,
    learning_rate=2e-4,
    bf16=True,
    logging_steps=10,
    gradient_checkpointing=True,
)

trainer = SFTTrainer(
    model=model,
    args=sft_config,
    train_dataset=dataset,
    processing_class=tokenizer,
    peft_config=lora_config,      # Pass config directly, SFTTrainer applies it
)

trainer.train()

Classification Task

from transformers import AutoModelForSequenceClassification
from peft import LoraConfig, get_peft_model, TaskType

model = AutoModelForSequenceClassification.from_pretrained(
    "bert-base-uncased",
    num_labels=2,
)

lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["query", "value"],
    lora_dropout=0.1,
    bias="none",
    task_type=TaskType.SEQ_CLS,
    modules_to_save=["classifier"],  # Train classification head fully
)

model = get_peft_model(model, lora_config)

Saving and Loading

Save Adapter

# Save only LoRA weights (small file)
model.save_pretrained("./my-lora-adapter")
tokenizer.save_pretrained("./my-lora-adapter")

# Push to Hub
model.push_to_hub("username/my-lora-adapter")

Load Adapter

from peft import PeftModel
from transformers import AutoModelForCausalLM

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-1B",
    dtype=torch.bfloat16,
    device_map="auto",
)

# Load adapter
model = PeftModel.from_pretrained(base_model, "./my-lora-adapter")

# For inference
model.eval()

Switch Between Adapters

# Load multiple adapters
model.load_adapter("./adapter-1", adapter_name="task1")
model.load_adapter("./adapter-2", adapter_name="task2")

# Switch active adapter
model.set_adapter("task1")
output = model.generate(**inputs)

model.set_adapter("task2")
output = model.generate(**inputs)

# Disable adapter (use base model)
with model.disable_adapter():
    output = model.generate(**inputs)

Per-Sample Adapter Selection

For inference batches that mix tasks or languages, pass adapter_names aligned with each sample. Use "__base__" for rows that should run the base model without an adapter.

inputs = tokenizer(["Hello", "Bonjour", "Hallo"], return_tensors="pt", padding=True).to(model.device)
adapter_names = ["__base__", "french", "german"]
outputs = model.generate(**inputs, adapter_names=adapter_names, max_new_tokens=32)

Merging Adapters

Merge LoRA weights into the base model for deployment without adapter overhead.

from peft import PeftModel

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-1B",
    dtype=torch.bfloat16,
    device_map="cpu",  # Merge on CPU to avoid memory issues
)

# Load adapter
model = PeftModel.from_pretrained(base_model, "./my-lora-adapter")

# Merge and unload
merged_model = model.merge_and_unload()

# Save merged model
merged_model.save_pretrained("./merged-model")
tokenizer.save_pretrained("./merged-model")

# Push merged model to Hub
merged_model.push_to_hub("username/my-merged-model")

Best Practices

  1. Start with r=16: Scale up to 32 or 64 if the model underfits, down to 8 if overfitting or memory-constrained

  2. Set lora_alpha = 2 × r: This is a common heuristic; the effective scaling is alpha/r

  3. Target all attention and MLP layers: For best results on LLMs, include gate/up/down projections:

    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
    
  4. Use higher learning rate: 2e-4 is typical for LoRA vs 2e-5 for full fine-tuning

  5. Enable gradient checkpointing: Reduces memory at cost of ~20% slower training:

    model.gradient_checkpointing_enable()
    
  6. Use QLoRA for large models: Essential for fine-tuning 7B+ models on consumer GPUs

  7. Keep dropout low: 0.05 is usually sufficient; higher values may hurt performance

  8. Save checkpoints frequently: LoRA adapters are small, so save often

  9. Evaluate on base model too: Ensure adapter doesn't degrade base capabilities

  10. Consider modules_to_save for task heads: For classification, train the classifier fully:

    modules_to_save=["classifier", "score"]
    
  11. Train new tokens selectively: After adding special tokens, prefer trainable_token_indices over training the whole embedding matrix when only a few tokens need adaptation.

  12. Merge adapters for high-throughput serving: merge_and_unload() removes PEFT runtime overhead, especially for MoE parameter-targeted adapters.

References

See reference/ for detailed documentation:

  • advanced-techniques.md - DoRA, rsLoRA, adapter composition, and debugging

External documentation:

Using MLX for LLMs on Apple Silicon

MLX-LM is a Python package for running large language models on Apple Silicon, leveraging the MLX framework for optimized performance with unified memory architecture.

Table of Contents

Core Concepts

Why MLX

AspectPyTorch on MacMLX
MemorySeparate CPU/GPU copiesUnified memory, no copies
OptimizationGeneric Metal backendApple Silicon native
Model loadingSlower, more memoryLazy loading, efficient
QuantizationLimited supportBuilt-in 4/8-bit

MLX arrays live in shared memory, accessible by both CPU and GPU without data transfer overhead.

Supported Models

MLX-LM supports most popular architectures: Llama, Mistral, Qwen, Phi, Gemma, Cohere, and many more. Check the mlx-community on Hugging Face for pre-converted models.

Installation

pip install mlx-lm

Requires macOS 13.5+ and Apple Silicon (M1/M2/M3/M4).

Text Generation

Python API

from mlx_lm import load, generate

# Load model (from HF hub or local path)
model, tokenizer = load("mlx-community/Llama-3.2-3B-Instruct-4bit")

# Generate text
response = generate(
    model,
    tokenizer,
    prompt="Explain quantum computing in simple terms:",
    max_tokens=256,
    temp=0.7,
)
print(response)

Streaming Generation

from mlx_lm import load, stream_generate

model, tokenizer = load("mlx-community/Mistral-7B-Instruct-v0.3-4bit")

prompt = "Write a haiku about programming:"
for response in stream_generate(model, tokenizer, prompt, max_tokens=100):
    print(response.text, end="", flush=True)
print()

Batch Generation

from mlx_lm import load, batch_generate

model, tokenizer = load("mlx-community/Qwen2.5-7B-Instruct-4bit")

prompts = [
    "What is machine learning?",
    "Explain neural networks:",
    "Define deep learning:",
]

responses = batch_generate(
    model,
    tokenizer,
    prompts,
    max_tokens=100,
)

for prompt, response in zip(prompts, responses):
    print(f"Q: {prompt}\nA: {response}\n")

CLI Generation

# Basic generation
mlx_lm.generate --model mlx-community/Llama-3.2-3B-Instruct-4bit \
    --prompt "Explain recursion:" \
    --max-tokens 256

# With sampling parameters
mlx_lm.generate --model mlx-community/Mistral-7B-Instruct-v0.3-4bit \
    --prompt "Write a poem about AI:" \
    --temp 0.8 \
    --top-p 0.95

Interactive Chat

CLI Chat

# Start chat REPL (context preserved between turns)
mlx_lm.chat --model mlx-community/Llama-3.2-3B-Instruct-4bit

Python Chat

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/Llama-3.2-3B-Instruct-4bit")

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What's the capital of France?"},
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

response = generate(model, tokenizer, prompt=prompt, max_tokens=256)
print(response)

Model Conversion

Convert Hugging Face models to MLX format:

CLI Conversion

# Convert with 4-bit quantization
mlx_lm.convert --model meta-llama/Llama-3.2-3B-Instruct \
    -q  # Quantize to 4-bit

# With specific quantization
mlx_lm.convert --model mistralai/Mistral-7B-Instruct-v0.3 \
    -q \
    --q-bits 8 \
    --q-group-size 64

# Upload to Hugging Face Hub
mlx_lm.convert --model meta-llama/Llama-3.2-1B-Instruct \
    -q \
    --upload-repo your-username/Llama-3.2-1B-Instruct-4bit-mlx

Python Conversion

from mlx_lm import convert

repo = "meta-llama/Llama-3.2-3B-Instruct"
convert(
    repo,
    quantize=True,
    mlx_path="./llama-3.2-3b-mlx",
    q_bits=4,
    q_group_size=64,
)

Conversion Options

OptionDefaultDescription
--q-bits4Quantization bits (4 or 8)
--q-group-size64Group size for quantization
--dtypefloat16Data type for non-quantized weights

Quantization

MLX supports multiple quantization methods for different use cases:

MethodBest ForCommand
BasicQuick conversionmlx_lm.convert -q
DWQQuality-preservingmlx_lm.dwq
AWQActivation-awaremlx_lm.awq
DynamicPer-layer precisionmlx_lm.dynamic_quant
GPTQEstablished methodmlx_lm.gptq

Quick Quantization

# 4-bit quantization during conversion
mlx_lm.convert --model mistralai/Mistral-7B-v0.3 -q

# 8-bit for higher quality
mlx_lm.convert --model mistralai/Mistral-7B-v0.3 -q --q-bits 8

For detailed coverage of each method, see reference/quantization.md.

Fine-tuning with LoRA

MLX supports LoRA and QLoRA fine-tuning for efficient adaptation on Apple Silicon.

Quick Start

pip install "mlx-lm[train]"

# Prepare training data (JSONL format)
# ./data/train.jsonl is required for --train
# ./data/valid.jsonl is optional
# Supported rows include {"text": "..."} and chat {"messages": [...]}

# Fine-tune with LoRA
mlx_lm.lora --model mlx-community/Llama-3.2-3B-Instruct-4bit \
    --train \
    --data ./data \
    --iters 1000

# Completion-only loss for chat/completion datasets
mlx_lm.lora --model mlx-community/Llama-3.2-3B-Instruct-4bit \
    --train \
    --data ./data \
    --mask-prompt

# Generate with adapter
mlx_lm.generate --model mlx-community/Llama-3.2-3B-Instruct-4bit \
    --adapter-path ./adapters \
    --prompt "Your prompt here"

Fuse Adapter into Model

# Merge LoRA weights into base model
mlx_lm.fuse --model mlx-community/Llama-3.2-3B-Instruct-4bit \
    --adapter-path ./adapters \
    --save-path ./fused-model

# Or export to GGUF
mlx_lm.fuse --model mlx-community/Llama-3.2-3B-Instruct-4bit \
    --adapter-path ./adapters \
    --export-gguf

For detailed LoRA configuration and training patterns, see reference/fine-tuning.md.

MLX-LM also supports --fine-tune-type dora and --fine-tune-type full when the task needs more capacity than standard LoRA.

Serving Models

OpenAI-Compatible Server

# Start server
mlx_lm.server --model mlx-community/Llama-3.2-3B-Instruct-4bit --port 8080

# Use with OpenAI client
curl http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "default",
        "messages": [{"role": "user", "content": "Hello!"}],
        "max_tokens": 256
    }'

Python Client

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="default",
    messages=[{"role": "user", "content": "Explain MLX in one sentence."}],
    max_tokens=100,
)
print(response.choices[0].message.content)

Best Practices

  1. Use pre-quantized models: Download from mlx-community on Hugging Face for immediate use

  2. Match quantization to your hardware: M1/M2 with 8GB: use 4-bit; M2/M3 Pro/Max: 8-bit for quality

  3. Leverage unified memory: Unlike CUDA, MLX models can exceed "GPU memory" by using swap (slower but works)

  4. Use streaming for UX: stream_generate provides responsive output for interactive applications

  5. Cache prompt prefixes: Use mlx_lm.cache_prompt for repeated prompts with varying suffixes

  6. Limit KV cache for long generations: Use --max-kv-size to cap memory at the cost of long-range quality

  7. Batch similar requests: batch_generate is more efficient than sequential generation

  8. Start with 4-bit quantization: Good quality/size tradeoff; upgrade to 8-bit if quality issues

  9. Fuse adapters for deployment: After fine-tuning, fuse adapters for faster inference without loading separately

  10. Monitor memory with Activity Monitor: Watch memory pressure to avoid swap thrashing

  11. Use chat templates: Always apply tokenizer.apply_chat_template() for instruction-tuned models

References

See reference/ for detailed documentation:

  • quantization.md - Detailed quantization methods and when to use each
  • fine-tuning.md - Complete LoRA/QLoRA training guide with data formats and configuration

External documentation:

Prompt Engineering

Prompt engineering is the practice of designing inputs that guide LLMs to produce desired outputs. Effective prompts reduce errors, improve consistency, and unlock model capabilities.

Table of Contents

Core Principles

Golden rule: Show your prompt to a colleague with minimal context. If they're confused, the model will be too.

  1. Be explicit - State exactly what you want; never assume the model knows your preferences
  2. Provide context - Include what the output is for, who the audience is, and what success looks like
  3. Use structure - Sequential steps, XML tags, and clear formatting reduce ambiguity
  4. Show examples - Demonstrations outperform descriptions for complex formats

Be Clear and Direct

Treat the model as a capable but context-free collaborator. Specify:

  • What the task results will be used for
  • What audience the output is meant for
  • What a successful completion looks like

Vague vs Specific

# Vague
Analyze this data and give insights.

# Specific
Analyze this Q2 sales data for our board presentation.
1. Identify the top 3 revenue trends
2. Flag any anomalies exceeding 15% variance
3. Recommend 2-3 actionable next steps
Format as bullet points, max 200 words.

Sequential Steps

Use numbered lists for multi-step tasks:

Your task is to anonymize customer feedback.

Instructions:
1. Replace customer names with "CUSTOMER_[ID]"
2. Replace emails with "EMAIL_[ID]@example.com"
3. Redact phone numbers as "PHONE_[ID]"
4. Leave product names intact
5. Output only processed messages, separated by "---"

Use Examples (Multishot)

Provide 3-5 diverse examples to demonstrate expected behavior. Examples reduce misinterpretation and enforce consistent formatting.

Structure

Categorize customer feedback by issue type and sentiment.

<examples>
<example>
Input: The dashboard loads slowly and the export button is hidden.
Category: UI/UX, Performance
Sentiment: Negative
Priority: High
</example>

<example>
Input: Love the Salesforce integration! Would be great to add Hubspot.
Category: Integration, Feature Request
Sentiment: Positive
Priority: Medium
</example>
</examples>

Now categorize: {{FEEDBACK}}

Tips

  • Make examples relevant to actual use cases
  • Include edge cases and potential challenges
  • Vary examples to prevent unintended pattern matching
  • Wrap in <example> tags for clarity

Reasoning Guidance

For complex tasks, guide the model to reason before answering, but ask for concise conclusions or verifiable work rather than unrestricted hidden chain-of-thought.

Basic

Determine the best investment option for this client. Think step-by-step.

Guided

Specify what steps to consider:

Think before answering:
1. Consider the client's risk tolerance given their 5-year timeline
2. Calculate potential returns for each option
3. Factor in market volatility history
4. Then provide your recommendation

Structured (Recommended)

Separate private analysis instructions from the required answer format:

Analyze this contract for legal risks.

Before answering, check:
- Indemnification implications
- Liability exposure
- IP ownership concerns

Then provide:
<answer>
- Top risks
- Recommended edits
- Open questions
</answer>

When you need auditability, ask for brief supporting rationale, calculations, cited evidence, or a checklist of checks performed. Do not require full chain-of-thought unless the target model or product explicitly supports exposing it.

XML Tags

Use XML tags to separate prompt components. This prevents instruction/content confusion and improves parseability.

Common Tags

<instructions>Task steps and requirements</instructions>
<context>Background information</context>
<document>Source material to process</document>
<example>Demonstration of expected behavior</example>
<constraints>Boundaries and limitations</constraints>
<output_format>Expected response structure</output_format>

Nested Structure

<documents>
  <document index="1">
    <source>annual_report_2023.pdf</source>
    <content>{{REPORT_CONTENT}}</content>
  </document>
  <document index="2">
    <source>competitor_analysis.xlsx</source>
    <content>{{ANALYSIS_CONTENT}}</content>
  </document>
</documents>

<instructions>
Compare revenue trends across both documents.
Identify strategic advantages mentioned in the annual report.
</instructions>

Reference Tags in Instructions

Be explicit when referring to tagged content:

Using the contract in <contract> tags, identify all clauses
related to termination.

Role Prompting

Set expertise context via system prompts to improve domain-specific performance.

System Prompt Pattern

system = "You are a senior securities lawyer at a Fortune 500 company."
user = "Review this acquisition agreement for regulatory risks."

Effective Roles

# General
You are a [role] at [organization type].

# Specific (better)
You are the General Counsel of a Fortune 500 tech company
specializing in M&A transactions.

# With behavioral guidance (best)
You are a senior data scientist. You prioritize statistical
rigor over speed. When uncertain, you state assumptions
explicitly and suggest validation approaches.

When to Use

  • Complex analysis requiring domain expertise
  • Tasks where tone/style matters (legal, medical, executive)
  • When a specific perspective would improve output quality

Long Context

For prompts with large documents (20K+ tokens):

Document Placement

Place long documents at the top, before instructions:

<documents>
{{LARGE_DOCUMENT_CONTENT}}
</documents>

<instructions>
Summarize the key findings from the document above.
Focus on financial implications.
</instructions>

Quote Grounding

Ask the model to cite sources before analyzing:

<documents>
{{PATIENT_RECORDS}}
</documents>

First, find and quote the relevant sections in <quotes> tags.
Then provide your diagnosis in <analysis> tags, referencing
the quoted evidence.

Multi-Document Metadata

Include source information for attribution:

<documents>
  <document index="1">
    <source>quarterly_report_q2.pdf</source>
    <date>2024-07-15</date>
    <content>{{CONTENT}}</content>
  </document>
</documents>

Output Control

Verbosity Specification

<output_format>
- Default responses: 3-6 sentences or ≤5 bullets
- Simple factual questions: ≤2 sentences
- Complex analysis: 1 overview paragraph + ≤5 tagged bullets
</output_format>

Format Constraints

Output requirements:
- Use markdown tables for comparisons
- Code blocks for any technical content
- No introductory phrases ("Here's...", "Sure...")
- End with exactly 3 action items

Scope Boundaries

Prevent drift from original intent:

Implement EXACTLY and ONLY what is requested.
- Do not add features beyond the specification
- Do not refactor surrounding code
- Choose the simplest valid interpretation
- Ask for clarification rather than assuming

Self-Verification

For high-stakes outputs, include verification steps:

<verification>
Before finalizing your response:
1. Re-read the original request
2. Check that all requirements are addressed
3. Verify any specific claims against provided documents
4. Soften language where certainty is low
5. Flag any assumptions you made
</verification>

Uncertainty Acknowledgment

When uncertain:
- Explicitly state "Based on the provided context..."
- Offer 2-3 plausible interpretations if ambiguous
- Never fabricate specific details (dates, numbers, quotes)
- Say "I don't have enough information to..." when applicable

Best Practices

  1. Start specific, then generalize - Begin with detailed prompts; relax constraints only after validating output quality
  2. Test with edge cases - Include unusual inputs in your evaluation to catch failure modes
  3. Iterate on examples - When outputs miss the mark, add an example demonstrating the correct behavior
  4. Separate instructions from content - Use XML tags to prevent the model from confusing your instructions with input data
  5. Put documents before queries - For long context, place source material at the top of the prompt
  6. Ask for verifiable reasoning artifacts - Prefer concise rationale, calculations, citations, or checklists over full hidden chain-of-thought
  7. Constrain output format explicitly - Specify structure, length, and style to reduce post-processing
  8. Version your prompts - Track changes to understand what modifications improved or degraded performance
  9. Use system prompts for role, user prompts for task - Keep role context stable; vary task instructions
  10. Validate with fresh eyes - Have someone unfamiliar with the task review your prompt for clarity

References

Using PyTorch

PyTorch is a deep learning framework with dynamic computation graphs, strong GPU acceleration, and Pythonic design. This skill covers practical patterns for building production-quality neural networks.

Table of Contents

Core Concepts

Tensors

import torch

# Create tensors
x = torch.tensor([[1, 2], [3, 4]], dtype=torch.float32)
x = torch.zeros(3, 4)
x = torch.randn(3, 4)  # Normal distribution

# Device management
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
x = x.to(device)

# Operations (all return new tensors)
y = x + 1
y = x @ x.T  # Matrix multiplication
y = x.view(2, 6)  # Reshape

Autograd

# Enable gradient tracking
x = torch.randn(3, requires_grad=True)
y = x ** 2
loss = y.sum()

# Compute gradients
loss.backward()
print(x.grad)  # dy/dx

# Disable gradients for inference
with torch.no_grad():
    pred = model(x)

# Or use inference mode (more efficient)
with torch.inference_mode():
    pred = model(x)

Model Architecture

nn.Module Pattern

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self, input_dim: int, hidden_dim: int, output_dim: int):
        super().__init__()
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        self.fc2 = nn.Linear(hidden_dim, output_dim)
        self.dropout = nn.Dropout(0.1)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = F.relu(self.fc1(x))
        x = self.dropout(x)
        return self.fc2(x)

Common Layers

# Convolution
nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1)

# Normalization
nn.BatchNorm2d(num_features)
nn.LayerNorm(normalized_shape)

# Attention
nn.MultiheadAttention(embed_dim, num_heads)

# Recurrent
nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)
nn.GRU(input_size, hidden_size, num_layers, batch_first=True)

Weight Initialization

def init_weights(module):
    if isinstance(module, nn.Linear):
        nn.init.xavier_uniform_(module.weight)
        if module.bias is not None:
            nn.init.zeros_(module.bias)
    elif isinstance(module, nn.Embedding):
        nn.init.normal_(module.weight, std=0.02)

model.apply(init_weights)

Training Loop

Standard Pattern

model = Model(input_dim, hidden_dim, output_dim).to(device)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)
criterion = nn.CrossEntropyLoss()
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_epochs)

for epoch in range(num_epochs):
    model.train()
    for batch in train_loader:
        inputs, targets = batch
        inputs, targets = inputs.to(device), targets.to(device)

        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, targets)
        loss.backward()

        # Optional: gradient clipping
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

        optimizer.step()

    scheduler.step()

    # Validation
    model.eval()
    with torch.no_grad():
        for batch in val_loader:
            # ... validation logic

Mixed Precision Training

from torch.amp import autocast, GradScaler

scaler = GradScaler("cuda")

for batch in train_loader:
    inputs, targets = batch
    inputs, targets = inputs.to(device), targets.to(device)

    optimizer.zero_grad()

    with autocast("cuda", dtype=torch.bfloat16):
        outputs = model(inputs)
        loss = criterion(outputs, targets)

    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()

Gradient Accumulation

# Requires setup from Mixed Precision Training above:
# scaler = GradScaler(), model, criterion, optimizer, device

accumulation_steps = 4

for i, batch in enumerate(train_loader):
    inputs, targets = batch
    inputs, targets = inputs.to(device), targets.to(device)

    with autocast("cuda", dtype=torch.bfloat16):
        outputs = model(inputs)
        loss = criterion(outputs, targets) / accumulation_steps

    scaler.scale(loss).backward()

    if (i + 1) % accumulation_steps == 0:
        scaler.step(optimizer)
        scaler.update()
        optimizer.zero_grad()

Data Loading

Dataset and DataLoader

from torch.utils.data import Dataset, DataLoader

class CustomDataset(Dataset):
    def __init__(self, data, labels, transform=None):
        self.data = data
        self.labels = labels
        self.transform = transform

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        x = self.data[idx]
        if self.transform:
            x = self.transform(x)
        return x, self.labels[idx]

train_loader = DataLoader(
    dataset,
    batch_size=32,
    shuffle=True,
    num_workers=4,
    pin_memory=True,  # Faster GPU transfer
    drop_last=True,   # Consistent batch sizes
)

Collate Functions

def collate_fn(batch):
    """Custom batching for variable-length sequences."""
    inputs, targets = zip(*batch)
    inputs = nn.utils.rnn.pad_sequence(inputs, batch_first=True)
    targets = torch.stack(targets)
    return inputs, targets

loader = DataLoader(dataset, collate_fn=collate_fn)

Performance Optimization

torch.compile (PyTorch 2.0+)

# Basic compilation
model = torch.compile(model)

# With options
model = torch.compile(
    model,
    mode="reduce-overhead",  # Options: default, reduce-overhead, max-autotune
    fullgraph=True,          # Enforce no graph breaks
)

# Compile specific functions
@torch.compile
def train_step(model, inputs, targets):
    outputs = model(inputs)
    return criterion(outputs, targets)

Compilation modes:

  • default: Good balance of compile time and speedup
  • reduce-overhead: Minimizes framework overhead, good for small models
  • max-autotune: Maximum performance, longer compile time

Start with fullgraph=False unless you are actively fixing graph breaks. Use TORCH_LOGS="graph_breaks" to inspect why compilation falls back to eager execution.

Memory Optimization

# Activation checkpointing (trade compute for memory)
from torch.utils.checkpoint import checkpoint

class Model(nn.Module):
    def forward(self, x):
        # Recompute activations during backward
        x = checkpoint(self.expensive_layer, x, use_reentrant=False)
        return self.output_layer(x)

# Clear cache
torch.cuda.empty_cache()

# Monitor memory
print(torch.cuda.memory_allocated() / 1e9, "GB")
print(torch.cuda.max_memory_allocated() / 1e9, "GB")

Distributed Training

DistributedDataParallel (DDP)

import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data.distributed import DistributedSampler

def setup(rank, world_size):
    dist.init_process_group("nccl", rank=rank, world_size=world_size)
    torch.cuda.set_device(rank)

def cleanup():
    dist.destroy_process_group()

def train(rank, world_size):
    setup(rank, world_size)

    model = Model().to(rank)
    model = DDP(model, device_ids=[rank])

    sampler = DistributedSampler(dataset, num_replicas=world_size, rank=rank)
    loader = DataLoader(dataset, sampler=sampler)

    for epoch in range(num_epochs):
        sampler.set_epoch(epoch)  # Important for shuffling
        # ... training loop

    cleanup()

# Launch with: torchrun --nproc_per_node=4 train.py

FullyShardedDataParallel (FSDP)

from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
from torch.distributed.fsdp import MixedPrecision

mp_policy = MixedPrecision(
    param_dtype=torch.bfloat16,
    reduce_dtype=torch.bfloat16,
    buffer_dtype=torch.bfloat16,
)

model = FSDP(
    model,
    mixed_precision=mp_policy,
    use_orig_params=True,  # Required for torch.compile compatibility
)

Saving and Loading

Checkpoints

# Save
torch.save({
    "epoch": epoch,
    "model_state_dict": model.state_dict(),
    "optimizer_state_dict": optimizer.state_dict(),
    "loss": loss,
}, "checkpoint.pt")

# Load
checkpoint = torch.load("checkpoint.pt", map_location=device, weights_only=True)
model.load_state_dict(checkpoint["model_state_dict"])
optimizer.load_state_dict(checkpoint["optimizer_state_dict"])

Export for Deployment

# TorchScript
scripted = torch.jit.script(model)
scripted.save("model.pt")

# ONNX
torch.onnx.export(
    model,
    dummy_input,
    "model.onnx",
    input_names=["input"],
    output_names=["output"],
    dynamic_axes={"input": {0: "batch"}, "output": {0: "batch"}},
)

Best Practices

  1. Always set model mode: Use model.train() and model.eval() appropriately

  2. Use inference_mode over no_grad: More efficient for inference

  3. Pin memory for GPU training: Set pin_memory=True in DataLoader

  4. Profile before optimizing: Use torch.profiler to find bottlenecks

  5. Prefer bfloat16 over float16: Better numerical stability on modern GPUs

  6. Use torch.compile: Significant speedups with minimal code changes

  7. Set deterministic mode for reproducibility:

    torch.manual_seed(42)
    torch.use_deterministic_algorithms(True)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    
  8. Use the unified AMP API: Prefer torch.amp.autocast("cuda") and torch.amp.GradScaler("cuda") over legacy torch.cuda.amp.

  9. Load untrusted checkpoints cautiously: Use weights_only=True when loading state-dict checkpoints.

References

See reference/ for detailed documentation:

  • training-patterns.md - Advanced training techniques
  • debugging.md - Debugging and profiling tools

External documentation:

QLoRA: Quantized Low-Rank Adaptation

QLoRA enables fine-tuning of large language models on consumer GPUs by combining 4-bit quantization with LoRA adapters. A 65B model can be fine-tuned on a single 48GB GPU while matching 16-bit fine-tuning performance.

Prerequisites: This skill assumes familiarity with LoRA. See the lora skill for LoRA fundamentals (LoraConfig, target_modules, training patterns).

Table of Contents

Core Innovations

QLoRA introduces three techniques that reduce memory usage without sacrificing performance:

4-bit NormalFloat (NF4)

NF4 is an information-theoretically optimal quantization data type for normally distributed weights. Neural network weights are typically normally distributed, making NF4 more efficient than standard 4-bit floats.

Storage: 4-bit NF4 (quantized weights)
Compute: 16-bit BF16 (dequantized for forward/backward pass)

The key insight: weights are stored in 4-bit but dequantized to bf16 for computation. Only the frozen base model is quantized; LoRA adapters remain in full precision.

NF4 vs FP4:

QuantizationDescriptionUse Case
nf4Normalized Float 4-bit, optimal for normal distributionsDefault, recommended
fp4Standard 4-bit floatLegacy, rarely needed

Double Quantization

Standard quantization requires storing scaling constants (typically fp32) for each quantization block. Double quantization quantizes these constants too:

First quantization:  weights → 4-bit + fp32 scaling constants
Double quantization: scaling constants → 8-bit + fp32 second-level constants

This saves approximately 0.37 bits per parameter—significant for billion-parameter models:

  • 7B model: ~325 MB savings
  • 70B model: ~3.2 GB savings

Paged Optimizers

During training, gradient checkpointing can cause memory spikes when processing long sequences. Paged optimizers use NVIDIA unified memory to automatically transfer optimizer states between GPU and CPU:

Normal training: OOM on memory spike
Paged optimizers: GPU ↔ CPU transfer handles spikes gracefully

This is handled automatically by bitsandbytes when using 4-bit training.

BitsAndBytesConfig Deep Dive

All Parameters Explained

from transformers import BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(
    # Core 4-bit settings
    load_in_4bit=True,              # Enable 4-bit quantization
    bnb_4bit_quant_type="nf4",      # "nf4" (recommended) or "fp4"

    # Double quantization
    bnb_4bit_use_double_quant=True, # Quantize the quantization constants

    # Compute precision
    bnb_4bit_compute_dtype=torch.bfloat16,  # Dequantize to this dtype for compute

    # Optional: specific storage type (usually auto-detected)
    bnb_4bit_quant_storage=torch.uint8,     # Storage dtype for quantized weights
)

Compute Dtype Selection

DtypeHardwareNotes
torch.bfloat16Ampere+ (RTX 30xx, A100)Recommended, faster
torch.float16Older GPUs (V100, RTX 20xx)Use if bf16 not supported
torch.float32AnySlower, only for debugging

Check bf16 support:

import torch
print(torch.cuda.is_bf16_supported())  # True on Ampere+

Comparison: Quantization Options

# Recommended: NF4 + double quant + bf16
optimal_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

# Fallback when bf16 is unsupported
fp16_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.float16,  # use when bf16 is unsupported or slower
)

# 8-bit alternative (less compression, sometimes more stable)
eight_bit_config = BitsAndBytesConfig(
    load_in_8bit=True,
)

Memory Requirements

Model SizeFull Fine-tuningLoRA (16-bit)QLoRA (4-bit)
7B~60 GB~16 GB~6 GB
13B~104 GB~28 GB~10 GB
34B~272 GB~75 GB~20 GB
70B~560 GB~160 GB~48 GB

Notes:

  • QLoRA memory includes model + optimizer states + activations
  • Actual usage varies with batch size, sequence length, and gradient checkpointing
  • Add ~20% buffer for safe operation

GPU Recommendations

GPU VRAMMax Model Size (QLoRA)
8 GB7B (tight)
16 GB7-13B
24 GB13-34B
48 GB34-70B
80 GB70B+ comfortably

Complete Training Example

from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    TrainingArguments,
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset
import torch

# 1. Quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

# 2. Load quantized model
model_name = "meta-llama/Llama-3.1-8B"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map="auto",
    dtype="auto",
    attn_implementation="flash_attention_2",  # Optional: faster attention
)

tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

# 3. Prepare for k-bit training (critical step!)
model = prepare_model_for_kbit_training(model)

# 4. LoRA config (see lora skill for parameter details)
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()

# 5. Dataset
dataset = load_dataset("tatsu-lab/alpaca", split="train[:1000]")

def format_example(example):
    if example["input"]:
        return {"text": f"### Instruction:\n{example['instruction']}\n\n### Input:\n{example['input']}\n\n### Response:\n{example['output']}"}
    return {"text": f"### Instruction:\n{example['instruction']}\n\n### Response:\n{example['output']}"}

dataset = dataset.map(format_example)

# 6. Training
sft_config = SFTConfig(
    output_dir="./qlora-output",
    max_length=512,
    dataset_text_field="text",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    num_train_epochs=1,
    learning_rate=2e-4,
    bf16=True,
    logging_steps=10,
    save_steps=100,
    gradient_checkpointing=True,
    gradient_checkpointing_kwargs={"use_reentrant": False},
    optim="paged_adamw_8bit",  # Paged optimizer for memory efficiency
)

trainer = SFTTrainer(
    model=model,
    args=sft_config,
    train_dataset=dataset,
    processing_class=tokenizer,
)

trainer.train()

# 7. Save adapter
model.save_pretrained("./qlora-adapter")
tokenizer.save_pretrained("./qlora-adapter")

Inference and Merging

Inference with Quantized Model

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch

model_name = "meta-llama/Llama-3.1-8B"

# Load quantized base model
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

base_model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map="auto",
    dtype="auto",
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Load adapter
model = PeftModel.from_pretrained(base_model, "./qlora-adapter")
model.eval()

# Generate
inputs = tokenizer("### Instruction:\nExplain quantum computing.\n\n### Response:\n", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Merging to Full Precision

To merge QLoRA adapters into a full-precision model (for deployment without bitsandbytes):

from transformers import AutoModelForCausalLM
from peft import PeftModel
import torch

# Load base model in full precision (on CPU to avoid OOM)
base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    dtype=torch.bfloat16,
    device_map="cpu",
)

# Load adapter
model = PeftModel.from_pretrained(base_model, "./qlora-adapter")

# Merge and unload
merged_model = model.merge_and_unload()

# Save merged model
merged_model.save_pretrained("./merged-model")

Note: Merging requires enough RAM to hold the full-precision model. For 70B models, this means ~140GB RAM.

Troubleshooting

CUDA Version Issues

# Check CUDA version
nvcc --version
python -c "import torch; print(torch.version.cuda)"

# bitsandbytes requires CUDA 11.7+
# If version mismatch, reinstall:
pip uninstall bitsandbytes
pip install bitsandbytes --upgrade

"cannot find libcudart" or Missing Library Errors

# Find CUDA installation
find /usr -name "libcudart*" 2>/dev/null

# Set environment variable
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

# Or for conda:
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH

Slow Training

Common cause: compute dtype mismatch

# Check if model is using expected dtype
for name, param in model.named_parameters():
    if param.requires_grad:
        print(f"{name}: {param.dtype}")
        break  # All LoRA params should match

# Ensure bf16 is used in training args if BitsAndBytesConfig uses bf16
# Mismatch causes constant dtype conversions

Out of Memory

# 1. Enable gradient checkpointing
model.gradient_checkpointing_enable()

# 2. Reduce batch size, increase accumulation
per_device_train_batch_size = 1
gradient_accumulation_steps = 16

# 3. Use paged optimizer
optim = "paged_adamw_8bit"

# 4. Reduce sequence length
max_length = 256

# 5. Target fewer modules
target_modules = ["q_proj", "v_proj"]  # Minimal set

Model Loads But Training Fails

# Ensure prepare_model_for_kbit_training is called
from peft import prepare_model_for_kbit_training
model = prepare_model_for_kbit_training(model)  # Don't skip this!

# Enable input gradients if needed
model.enable_input_require_grads()

Best Practices

  1. Always use prepare_model_for_kbit_training: This enables gradient computation through the frozen quantized layers

  2. Match compute dtype with training precision: If bnb_4bit_compute_dtype=torch.bfloat16, use bf16=True in training args

  3. Use paged optimizers for large models: optim="paged_adamw_8bit" or "paged_adamw_32bit" handles memory spikes

  4. Start with NF4 + double quantization: This is the recommended default; only change if debugging

  5. Gradient checkpointing is essential: Always enable for QLoRA training to fit larger batch sizes

  6. Test inference before long training runs: Load the model and generate a few tokens to catch configuration issues early

  7. Monitor GPU memory: Use nvidia-smi or torch.cuda.memory_summary() to track actual usage

  8. Consider 8-bit for unstable training: If 4-bit training shows instability, try load_in_8bit=True as a middle ground

  9. Use current TRL config fields: Put max_length and dataset_text_field in SFTConfig, and pass the tokenizer or processor as processing_class.

References

Understanding RLHF

Reinforcement Learning from Human Feedback (RLHF) is a technique for aligning language models with human preferences. Rather than relying solely on next-token prediction, RLHF uses human judgment to guide model behavior toward helpful, harmless, and honest outputs.

Table of Contents

Core Concepts

Why RLHF?

Pretraining produces models that predict likely text, not necessarily good text. A model trained on internet data learns to complete text in ways that reflect its training distribution—including toxic, unhelpful, or dishonest patterns. RLHF addresses this gap by optimizing for human preferences rather than likelihood.

The core insight: humans can often recognize good outputs more easily than they can specify what makes an output good. RLHF exploits this by collecting human judgments and using them to shape model behavior.

The Alignment Problem

Language models face several alignment challenges:

  • Helpfulness: Following instructions and providing useful information
  • Harmlessness: Avoiding toxic, dangerous, or inappropriate outputs
  • Honesty: Acknowledging uncertainty and avoiding fabrication
  • Intent alignment: Understanding what users actually want, not just what they say

RLHF provides a framework for encoding these properties through preference data.

Key Components

  1. Preference data: Human judgments comparing model outputs
  2. Reward model: A learned function approximating human preferences
  3. Policy optimization: RL algorithms that maximize expected reward
  4. Regularization: Constraints preventing deviation from the base model

The RLHF Pipeline

The standard RLHF pipeline consists of three main stages:

Stage 1: Supervised Fine-Tuning (SFT)

Start with a pretrained language model and fine-tune it on high-quality demonstrations. This teaches the model the desired format and style of responses.

Input: Pretrained model + demonstration dataset Output: SFT model that can follow instructions

Stage 2: Reward Model Training

Train a model to predict human preferences between pairs of outputs. The reward model learns to score outputs in a way that correlates with human judgment.

Input: SFT model + preference dataset (chosen/rejected pairs) Output: Reward model that scores any output

Stage 3: Policy Optimization

Use reinforcement learning to optimize the SFT model against the reward model, while staying close to the original SFT distribution.

Input: SFT model + reward model Output: Final aligned model

Alternative: Direct Alignment

Direct alignment algorithms (DPO, IPO, KTO) skip the reward model entirely, optimizing directly from preference data. This simplifies the pipeline but trades off some flexibility.

Modern Post-Training Variants

Current post-training stacks often mix these stages rather than using a single linear pipeline:

MethodData SignalBest Fit
SFTDemonstrationsFormat, style, instruction following
DPO/IPO/KTO/ORPOOffline preferences or binary feedbackSimpler alignment without online rollouts
PPO/RLOOReward model scores on sampled responsesReward-model RL with explicit KL control
GRPOGrouped completions scored by reward functions/modelsReasoning and verifiable-task optimization

Preference Data

Preference data encodes human judgment about model outputs. The most common format is pairwise comparisons.

Pairwise Preferences

Given a prompt, collect two or more model outputs and have humans indicate which is better:

Prompt: "Explain quantum entanglement"

Response A: [technical explanation]
Response B: [simpler explanation with analogy]

Human preference: B > A

This creates (prompt, chosen, rejected) tuples for training.

Collection Methods

Human annotation: Trained annotators compare outputs according to guidelines. Most reliable but expensive and slow.

AI feedback: Use a capable model to generate preferences. Faster and cheaper but may propagate biases. This is the basis for Constitutional AI (CAI) and RLAIF.

Implicit signals: User interactions like upvotes, regeneration requests, or conversation length. Noisy but abundant.

Data Quality Considerations

  • Annotator agreement: Low agreement suggests ambiguous criteria or subjective preferences
  • Distribution coverage: Preferences should cover the range of model behaviors
  • Prompt diversity: Diverse prompts prevent overfitting to narrow scenarios
  • Preference strength: Some comparisons are clear; others are nearly ties

Instruction Tuning

Instruction tuning (supervised fine-tuning on instruction-response pairs) serves as the foundation for RLHF.

Purpose

  • Teaches the model to follow instructions rather than just complete text
  • Establishes the format and style for responses
  • Creates a starting point that already exhibits desired behaviors
  • Provides the reference policy for KL regularization

Dataset Composition

Typical instruction tuning datasets include:

  • Single-turn QA: Questions with direct answers
  • Multi-turn dialogue: Conversational exchanges
  • Task instructions: Specific tasks with examples
  • Chain-of-thought: Reasoning traces for complex problems

Relationship to RLHF

The SFT model defines the "prior" that RLHF refines. A better SFT model means:

  • The reward model has better starting outputs to compare
  • Policy optimization has less work to do
  • The KL penalty keeps the final model closer to this baseline

Reward Modeling

The reward model transforms pairwise preferences into a scalar signal for RL optimization.

The Bradley-Terry Model

Preferences are modeled using the Bradley-Terry framework:

P(A > B) = sigmoid(r(A) - r(B))

Where r(x) is the reward for output x. This assumes preferences depend only on the difference in rewards.

The loss function is:

L = -log(sigmoid(r(chosen) - r(rejected)))

This pushes the reward model to assign higher scores to chosen outputs.

Architecture

Reward models are typically:

  • The SFT model with a scalar head instead of the language modeling head
  • Trained on (prompt, chosen, rejected) tuples
  • Output a single scalar reward for any (prompt, response) pair

Considerations

  • Scaling: Larger reward models generally produce better signals
  • Calibration: Absolute reward values are less important than rankings
  • Generalization: The model must score outputs it hasn't seen during training
  • Over-optimization: Policies can exploit reward model weaknesses

See reference/reward-modeling.md for detailed training procedures.

Policy Optimization

Policy optimization uses RL to maximize expected reward while staying close to the reference policy.

The RLHF Objective

maximize E[R(x, y)] - β * KL(π || π_ref)

Where:

  • R(x, y) is the reward for response y to prompt x
  • KL(π || π_ref) measures deviation from the reference policy
  • β controls the strength of the regularization

PPO (Proximal Policy Optimization)

PPO is the classic reward-model RLHF algorithm:

  1. Sample responses from the current policy
  2. Score responses with the reward model
  3. Compute advantage estimates
  4. Update policy with clipped surrogate objective

The clipping prevents large policy updates that could destabilize training.

KL Regularization

The KL penalty serves multiple purposes:

  • Prevents reward hacking: Stops the policy from finding adversarial inputs to the reward model
  • Maintains capabilities: Keeps the model close to the pretrained distribution
  • Stabilizes training: Limits how far the policy can move per update

Higher β means more conservative updates; lower β allows more aggressive optimization.

REINFORCE vs PPO

REINFORCE is simpler but has higher variance:

  • Uses raw returns without value function baseline
  • Single-sample gradient estimates
  • Can work for simpler problems

PPO adds complexity but improves stability:

  • Clipped surrogate objective
  • Multiple epochs per batch
  • Better sample efficiency

See reference/policy-optimization.md for algorithm details.

GRPO (Group Relative Policy Optimization)

GRPO samples multiple completions for each prompt, scores them with reward functions or reward models, and normalizes advantages within the group. It removes the separate value model used by PPO, which can reduce memory and implementation complexity for LLM post-training.

Use GRPO when:

  • Rewards can be computed automatically, such as math correctness, unit tests, format checks, or model-graded rubrics
  • You want online learning from the model's own sampled completions
  • Memory pressure makes PPO's value model unattractive

Watch for:

  • Reward scaling choices, which can bias learning toward easier or shorter prompts
  • Completion length bias
  • Reward functions that are brittle or easy to exploit

Direct Alignment Algorithms

Direct alignment methods optimize the RLHF objective without training a separate reward model.

DPO (Direct Preference Optimization)

DPO reparameterizes the RLHF objective to derive a closed-form loss:

L = -log sigmoid(β * (log π(y_w|x)/π_ref(y_w|x) - log π(y_l|x)/π_ref(y_l|x)))

Where y_w is the preferred response and y_l is the dispreferred response.

Advantages:

  • No separate reward model training
  • Simpler pipeline with fewer hyperparameters
  • More stable than RL-based methods

Trade-offs:

  • Less flexible than explicit reward models
  • Cannot reuse reward model for other purposes
  • May be more sensitive to data quality

IPO (Identity Preference Optimization)

IPO addresses potential overfitting in DPO by using a different loss formulation that doesn't assume the Bradley-Terry model perfectly describes preferences.

KTO (Kahneman-Tversky Optimization)

KTO works with binary feedback (good/bad) rather than pairwise comparisons, making data collection easier. It's based on prospect theory from behavioral economics.

When to Use Direct Alignment

Direct alignment is preferred when:

  • Simplicity is important
  • Computational resources are limited
  • The reward model won't be reused

Reward-based RLHF is preferred when:

  • You need the reward model for other purposes (filtering, ranking)
  • You have a large preference dataset
  • You want maximum flexibility in optimization

See reference/direct-alignment.md for detailed algorithm comparisons.

Challenges

Over-Optimization

As optimization proceeds, the policy may exploit weaknesses in the reward model rather than improving on the true objective. Symptoms include:

  • Rising reward model scores but declining human evaluation
  • Increasingly verbose or formulaic outputs
  • Sycophantic behavior (agreeing with users regardless of correctness)

Mitigations:

  • Stronger KL regularization
  • Reward model ensembles
  • Early stopping based on held-out evaluation

Reward Hacking

The policy finds inputs that score highly with the reward model but don't represent genuine improvement:

  • Length exploitation (longer responses score higher)
  • Style mimicry (copying patterns from high-reward examples)
  • Adversarial outputs that confuse the reward model

Evaluation

Evaluating aligned models is difficult because:

  • Human preferences are subjective and context-dependent
  • Automated metrics don't capture alignment properties well
  • A/B testing is expensive and slow
  • Models may perform differently on evaluation vs deployment

Distribution Shift

The preference data comes from a specific distribution of prompts and responses. The deployed model will encounter different inputs, and the reward model may not generalize well.

Best Practices

  1. Start with a strong SFT model: RLHF refines behavior; it works best when the base model already exhibits desired patterns
  2. Invest in preference data quality: Garbage in, garbage out—clear guidelines and trained annotators matter
  3. Use KL regularization: Don't optimize reward too aggressively; the reward model is an imperfect proxy
  4. Monitor for reward hacking: Track human evaluations alongside reward model scores
  5. Consider direct alignment first: DPO/ORPO-style methods are simpler than online RL when offline preference data is enough
  6. Iterate on reward model: Improve the reward model as you discover its weaknesses
  7. Diverse prompts: Ensure preference data covers the distribution you care about
  8. Regularize appropriately: Higher β for safety-critical applications; lower β for capability-focused training
  9. Use GRPO for verifiable reasoning tasks: Prefer reward functions with objective checks before model-graded rubrics

References

Reference Files

  • reference/reward-modeling.md - Detailed reward model training procedures
  • reference/policy-optimization.md - PPO and policy gradient algorithms for RLHF
  • reference/direct-alignment.md - DPO, IPO, KTO and other direct methods

External Resources

Using Hugging Face Transformers

Transformers is the model-definition framework for state-of-the-art machine learning across text, vision, audio, and multimodal domains. It provides unified APIs for loading pretrained models, running inference, and fine-tuning.

Table of Contents

Core Concepts

The Three Core Classes

Every model in Transformers has three core components:

from transformers import AutoConfig, AutoModel, AutoTokenizer, AutoProcessor

# Configuration: hyperparameters and architecture settings
config = AutoConfig.from_pretrained("bert-base-uncased")

# Model: the neural network weights
model = AutoModel.from_pretrained("bert-base-uncased")

# Tokenizer: converts text inputs to tensors
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

# Processor: unified preprocessing for vision, audio, and multimodal models
processor = AutoProcessor.from_pretrained("openai/whisper-large-v3")

The from_pretrained Pattern

All loading uses from_pretrained() which handles downloading, caching, and device placement:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "meta-llama/Llama-3.2-1B"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    dtype=torch.bfloat16,
    device_map="auto",  # Automatic device placement
)

Transformers v5 examples use dtype. On Transformers v4, the equivalent argument is torch_dtype.

Auto Classes

Use task-specific Auto classes for the correct model head:

from transformers import (
    AutoModelForCausalLM,          # Text generation (GPT, Llama)
    AutoModelForSeq2SeqLM,         # Encoder-decoder (T5, BART)
    AutoModelForSequenceClassification,  # Classification
    AutoModelForTokenClassification,     # NER, POS tagging
    AutoModelForQuestionAnswering,       # Extractive QA
    AutoModelForMaskedLM,                # BERT-style masked LM
    AutoModelForImageClassification,     # Vision models
    AutoModelForSpeechSeq2Seq,           # Speech recognition
)

Pipeline API

The pipeline() function provides high-level inference with minimal code:

Text Tasks

from transformers import pipeline

# Text generation
generator = pipeline("text-generation", model="Qwen/Qwen2.5-1.5B")
output = generator("The secret to success is", max_new_tokens=50)

# Text classification
classifier = pipeline("sentiment-analysis")
result = classifier("I love this product!")
# [{'label': 'POSITIVE', 'score': 0.9998}]

# Named entity recognition
ner = pipeline("ner", aggregation_strategy="simple")
entities = ner("Hugging Face is based in New York City.")

# Question answering
qa = pipeline("question-answering")
answer = qa(question="What is the capital?", context="Paris is the capital of France.")

# Summarization
summarizer = pipeline("summarization", model="facebook/bart-large-cnn")
summary = summarizer(long_text, max_length=130, min_length=30)

# Translation
translator = pipeline("translation_en_to_fr", model="Helsinki-NLP/opus-mt-en-fr")
result = translator("Hello, how are you?")

Chat/Conversational

from transformers import pipeline
import torch

pipe = pipeline(
    "text-generation",
    model="meta-llama/Llama-3.2-3B-Instruct",
    dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain quantum computing in simple terms."},
]

response = pipe(messages, max_new_tokens=256)
print(response[0]["generated_text"][-1]["content"])

Vision Tasks

classifier = pipeline("image-classification", model="google/vit-base-patch16-224")
detector = pipeline("object-detection", model="facebook/detr-resnet-50")

Audio Tasks

transcriber = pipeline("automatic-speech-recognition", model="openai/whisper-large-v3")
text = transcriber("path/to/audio.mp3")

Multimodal Tasks

vqa = pipeline("visual-question-answering", model="Salesforce/blip-vqa-base")
captioner = pipeline("image-to-text", model="Salesforce/blip-image-captioning-base")

Model Loading

Device Placement

from transformers import AutoModelForCausalLM
import torch

# Automatic placement across available devices
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B",
    device_map="auto",
    dtype=torch.bfloat16,
)

# Specific device
model = AutoModelForCausalLM.from_pretrained(
    "gpt2",
    device_map="cuda:0",
)

# Custom device map for model parallelism
device_map = {
    "model.embed_tokens": 0,
    "model.layers.0": 0,
    "model.layers.1": 1,
    "model.norm": 1,
    "lm_head": 1,
}
model = AutoModelForCausalLM.from_pretrained(model_name, device_map=device_map)

Loading from Local Path

# Save model locally
model.save_pretrained("./my_model")
tokenizer.save_pretrained("./my_model")

# Load from local path
model = AutoModelForCausalLM.from_pretrained("./my_model")
tokenizer = AutoTokenizer.from_pretrained("./my_model")

Trust Remote Code

Some models require executing custom code from the Hub:

model = AutoModelForCausalLM.from_pretrained(
    "microsoft/phi-2",
    trust_remote_code=True,  # Required for custom architectures
)

Prefer models with safetensors weights when available. Safetensors avoids pickle execution risks and typically loads faster than legacy .bin checkpoints.

Inference Patterns

Text Generation

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "Qwen/Qwen2.5-3B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    dtype=torch.bfloat16,
    device_map="auto",
)

# Basic generation
inputs = tokenizer("Once upon a time", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
text = tokenizer.decode(outputs[0], skip_special_tokens=True)

# With generation config
outputs = model.generate(
    **inputs,
    max_new_tokens=100,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    top_k=50,
    repetition_penalty=1.1,
)

Chat Templates

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the capital of France?"},
]

# Apply chat template
input_text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)

Getting Embeddings

from transformers import AutoModel, AutoTokenizer
import torch

model = AutoModel.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")
tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")

def get_embeddings(texts: list[str]) -> torch.Tensor:
    inputs = tokenizer(texts, padding=True, truncation=True, return_tensors="pt")

    with torch.no_grad():
        outputs = model(**inputs)

    # Mean pooling
    attention_mask = inputs["attention_mask"]
    embeddings = outputs.last_hidden_state
    mask_expanded = attention_mask.unsqueeze(-1).expand(embeddings.size()).float()
    sum_embeddings = (embeddings * mask_expanded).sum(1)
    sum_mask = mask_expanded.sum(1).clamp(min=1e-9)
    return sum_embeddings / sum_mask

embeddings = get_embeddings(["Hello world", "How are you?"])

Classification

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english")
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english")

inputs = tokenizer("I love this movie!", return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)
    predictions = torch.softmax(outputs.logits, dim=-1)

labels = model.config.id2label
for idx, prob in enumerate(predictions[0]):
    print(f"{labels[idx]}: {prob:.4f}")

Fine-tuning with Trainer

Basic Fine-tuning

from transformers import (
    AutoModelForSequenceClassification,
    AutoTokenizer,
    Trainer,
    TrainingArguments,
)
from datasets import load_dataset

# Load data and model
dataset = load_dataset("imdb")
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
model = AutoModelForSequenceClassification.from_pretrained(
    "distilbert-base-uncased",
    num_labels=2,
)

# Tokenize dataset
def tokenize(examples):
    return tokenizer(examples["text"], padding="max_length", truncation=True)

tokenized = dataset.map(tokenize, batched=True)

# Training arguments
training_args = TrainingArguments(
    output_dir="./results",
    eval_strategy="epoch",
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    num_train_epochs=3,
    weight_decay=0.01,
    logging_steps=100,
    save_strategy="epoch",
    load_best_model_at_end=True,
)

# Train
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized["train"],
    eval_dataset=tokenized["test"],
)

trainer.train()

Pushing to Hub

# Login first: huggingface-cli login

# Push model and tokenizer
model.push_to_hub("my-username/my-fine-tuned-model")
tokenizer.push_to_hub("my-username/my-fine-tuned-model")

# Or use trainer
trainer.push_to_hub()

See reference/fine-tuning.md for advanced patterns including LoRA, custom data collators, and evaluation metrics.

Working with Modalities

Use AutoProcessor or modality-specific processors for non-text models. Processors handle images, audio, video, and multimodal chat formatting before tensors are sent to the model.

ModalityProcessorModel Class
VisionAutoImageProcessorAutoModelForImageClassification, AutoModelForObjectDetection
AudioAutoProcessorAutoModelForSpeechSeq2Seq, AutoModelForAudioClassification
Vision-languageAutoProcessorAutoModelForVision2Seq, task-specific VLM classes

Memory and Performance

Quantization

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

# 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B",
    quantization_config=bnb_config,
    device_map="auto",
    dtype="auto",
)

# 8-bit quantization
bnb_config = BitsAndBytesConfig(load_in_8bit=True)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B",
    quantization_config=bnb_config,
    device_map="auto",
    dtype="auto",
)

Flash Attention

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B",
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",  # Requires flash-attn package
    device_map="auto",
)

torch.compile

model = AutoModelForCausalLM.from_pretrained(model_name, dtype=torch.bfloat16)
model = torch.compile(model, mode="reduce-overhead")

Batched Inference

texts = ["First prompt", "Second prompt", "Third prompt"]
inputs = tokenizer(texts, return_tensors="pt", padding=True).to(model.device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=50)

decoded = tokenizer.batch_decode(outputs, skip_special_tokens=True)

Best Practices

  1. Use bfloat16 over float16: Better numerical stability on modern GPUs
  2. Use dtype="auto" when unsure: It follows model metadata or checkpoint weight dtype without wasting memory
  3. Set pad token for generation: tokenizer.pad_token = tokenizer.eos_token
  4. Use device_map="auto": Let Accelerate handle device placement
  5. Prefer safetensors checkpoints: Avoid pickle-based loading when possible
  6. Enable Flash Attention: Significant speedup for long sequences
  7. Batch when possible: Amortize fixed costs across multiple inputs
  8. Use pipeline for quick prototyping: Switch to manual control for production
  9. Cache models locally: Set HF_HOME environment variable for model cache location
  10. Check model license: Verify usage rights before deployment

References

See reference/ for detailed documentation:

  • fine-tuning.md - Advanced fine-tuning patterns with LoRA, PEFT, and custom training

External documentation:

Sponsors

Details

Marketplace
itsmostafa/llm-engineering-skills
Category
Community