
security-fuzzing
Fuzzing, sanitizers, property-based and cryptographic testing
- version
- 1.0.0
- GitHub stars
- 5
Install
/plugin install security-fuzzing@skills-stackRun it inside Claude Code after adding the marketplace with /plugin marketplace add marslanalvi/SkillsStack.
Contents
Discovered from sourceSkills 19
AddressSanitizer (ASan)
AddressSanitizer (ASan) is a widely adopted memory error detection tool used extensively during software testing, particularly fuzzing. It helps detect memory corruption bugs that might otherwise go unnoticed, such as buffer overflows, use-after-free errors, and other memory safety violations.
Overview
ASan is a standard practice in fuzzing due to its effectiveness in identifying memory vulnerabilities. It instruments code at compile time to track memory allocations and accesses, detecting illegal operations at runtime.
Key Concepts
| Concept | Description |
|---|---|
| Instrumentation | ASan adds runtime checks to memory operations during compilation |
| Shadow Memory | Maps 20TB of virtual memory to track allocation state |
| Performance Cost | Approximately 2-4x slowdown compared to non-instrumented code |
| Detection Scope | Finds buffer overflows, use-after-free, double-free, and memory leaks |
When to Apply
Apply this technique when:
- Fuzzing C/C++ code for memory safety vulnerabilities
- Testing Rust code with unsafe blocks
- Debugging crashes related to memory corruption
- Running unit tests where memory errors are suspected
Skip this technique when:
- Running production code (ASan can reduce security)
- Platform is Windows or macOS (limited ASan support)
- Performance overhead is unacceptable for your use case
- Fuzzing pure safe languages without FFI (e.g., pure Go, pure Java)
Quick Reference
| Task | Command/Pattern |
|---|---|
| Enable ASan (Clang/GCC) | -fsanitize=address |
| Enable verbosity | ASAN_OPTIONS=verbosity=1 |
| Disable leak detection | ASAN_OPTIONS=detect_leaks=0 |
| Force abort on error | ASAN_OPTIONS=abort_on_error=1 |
| Multiple options | ASAN_OPTIONS=verbosity=1:abort_on_error=1 |
Step-by-Step
Step 1: Compile with ASan
Compile and link your code with the -fsanitize=address flag:
clang -fsanitize=address -g -o my_program my_program.c
The -g flag is recommended to get better stack traces when ASan detects errors.
Step 2: Configure ASan Options
Set the ASAN_OPTIONS environment variable to configure ASan behavior:
export ASAN_OPTIONS=verbosity=1:abort_on_error=1:detect_leaks=0
Step 3: Run Your Program
Execute the ASan-instrumented binary. When memory errors are detected, ASan will print detailed reports:
./my_program
Step 4: Adjust Fuzzer Memory Limits
ASan requires approximately 20TB of virtual memory. Disable fuzzer memory restrictions:
- libFuzzer:
-rss_limit_mb=0 - AFL++:
-m none
Common Patterns
Pattern: Basic ASan Integration
Use Case: Standard fuzzing setup with ASan
Before:
clang -o fuzz_target fuzz_target.c
./fuzz_target
After:
clang -fsanitize=address -g -o fuzz_target fuzz_target.c
ASAN_OPTIONS=verbosity=1:abort_on_error=1 ./fuzz_target
Pattern: ASan with Unit Tests
Use Case: Enable ASan for unit test suite
Before:
gcc -o test_suite test_suite.c -lcheck
./test_suite
After:
gcc -fsanitize=address -g -o test_suite test_suite.c -lcheck
ASAN_OPTIONS=detect_leaks=1 ./test_suite
Advanced Usage
Tips and Tricks
| Tip | Why It Helps |
|---|---|
Use -g flag | Provides detailed stack traces for debugging |
Set verbosity=1 | Confirms ASan is enabled before program starts |
| Disable leaks during fuzzing | Leak detection doesn't cause immediate crashes, clutters output |
Enable abort_on_error=1 | Some fuzzers require abort() instead of _exit() |
Understanding ASan Reports
When ASan detects a memory error, it prints a detailed report including:
- Error type: Buffer overflow, use-after-free, etc.
- Stack trace: Where the error occurred
- Allocation/deallocation traces: Where memory was allocated/freed
- Memory map: Shadow memory state around the error
Example ASan report:
==12345==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60300000eff4 at pc 0x00000048e6a3
READ of size 4 at 0x60300000eff4 thread T0
#0 0x48e6a2 in main /path/to/file.c:42
Combining Sanitizers
ASan can be combined with other sanitizers for comprehensive detection:
clang -fsanitize=address,undefined -g -o fuzz_target fuzz_target.c
Platform-Specific Considerations
Linux: Full ASan support with best performance macOS: Limited support, some features may not work Windows: Experimental support, not recommended for production fuzzing
Anti-Patterns
| Anti-Pattern | Problem | Correct Approach |
|---|---|---|
| Using ASan in production | Can make applications less secure | Use ASan only for testing |
| Not disabling memory limits | Fuzzer may kill process due to 20TB virtual memory | Set -rss_limit_mb=0 or -m none |
| Ignoring leak reports | Memory leaks indicate resource management issues | Review leak reports at end of fuzzing campaign |
Tool-Specific Guidance
libFuzzer
Compile with both fuzzer and address sanitizer:
clang++ -fsanitize=fuzzer,address -g harness.cc -o fuzz
Run with unlimited RSS:
./fuzz -rss_limit_mb=0
Integration tips:
- Always combine
-fsanitize=fuzzerwith-fsanitize=address - Use
-gfor detailed stack traces in crash reports - Consider
ASAN_OPTIONS=abort_on_error=1for better crash handling
See: libFuzzer: AddressSanitizer
AFL++
Use the AFL_USE_ASAN environment variable:
AFL_USE_ASAN=1 afl-clang-fast++ -g harness.cc -o fuzz
Run with unlimited memory:
afl-fuzz -m none -i input_dir -o output_dir ./fuzz
Integration tips:
AFL_USE_ASAN=1automatically adds proper compilation flags- Use
-m noneto disable AFL++'s memory limit - Consider
AFL_MAP_SIZEfor programs with large coverage maps
cargo-fuzz (Rust)
Use the --sanitizer=address flag:
cargo fuzz run fuzz_target --sanitizer=address
Or configure in fuzz/Cargo.toml:
[profile.release]
opt-level = 3
debug = true
Integration tips:
- ASan is useful for fuzzing unsafe Rust code or FFI boundaries
- Safe Rust code may not benefit as much (compiler already prevents many errors)
- Focus on unsafe blocks, raw pointers, and C library bindings
See: cargo-fuzz: AddressSanitizer
honggfuzz
Compile with ASan and link with honggfuzz:
honggfuzz -i input_dir -o output_dir -- ./fuzz_target_asan
Compile the target:
hfuzz-clang -fsanitize=address -g target.c -o fuzz_target_asan
Integration tips:
- honggfuzz works well with ASan out of the box
- Use feedback-driven mode for better coverage with sanitizers
- Monitor memory usage, as ASan increases memory footprint
Troubleshooting
| Issue | Cause | Solution |
|---|---|---|
| Fuzzer kills process immediately | Memory limit too low for ASan's 20TB virtual memory | Use -rss_limit_mb=0 (libFuzzer) or -m none (AFL++) |
| "ASan runtime not initialized" | Wrong linking order or missing runtime | Ensure -fsanitize=address used in both compile and link |
| Leak reports clutter output | LeakSanitizer enabled by default | Set ASAN_OPTIONS=detect_leaks=0 |
| Poor performance (>4x slowdown) | Debug mode or unoptimized build | Compile with -O2 or -O3 alongside -fsanitize=address |
| ASan not detecting obvious bugs | Binary not instrumented | Check with ASAN_OPTIONS=verbosity=1 that ASan prints startup info |
| False positives | Interceptor conflicts | Check ASan FAQ for known issues with specific libraries |
Related Skills
Tools That Use This Technique
| Skill | How It Applies |
|---|---|
| libfuzzer | Compile with -fsanitize=fuzzer,address for integrated fuzzing with memory error detection |
| aflpp | Use AFL_USE_ASAN=1 environment variable during compilation |
| cargo-fuzz | Use --sanitizer=address flag to enable ASan for Rust fuzz targets |
| honggfuzz | Compile target with -fsanitize=address for ASan-instrumented fuzzing |
Related Techniques
| Skill | Relationship |
|---|---|
| undefined-behavior-sanitizer | Often used together with ASan for comprehensive bug detection (undefined behavior + memory errors) |
| fuzz-harness-writing | Harnesses must be designed to handle ASan-detected crashes and avoid false positives |
| coverage-analysis | Coverage-guided fuzzing helps trigger code paths where ASan can detect memory errors |
Resources
Key External Resources
AddressSanitizer on Google Sanitizers Wiki
The official ASan documentation covers:
- Algorithm and implementation details
- Complete list of detected error types
- Performance characteristics and overhead
- Platform-specific behavior
- Known limitations and incompatibilities
Common configuration flags shared across all sanitizers:
verbosity: Control diagnostic output levellog_path: Redirect sanitizer output to filessymbolize: Enable/disable symbol resolution in reportsexternal_symbolizer_path: Use custom symbolizer
ASan-specific configuration options:
detect_leaks: Control memory leak detectionabort_on_error: Callabort()vs_exit()on errordetect_stack_use_after_return: Detect stack use-after-return bugscheck_initialization_order: Find initialization order bugs
Common pitfalls and solutions:
- Linking order issues
- Conflicts with other tools
- Platform-specific problems
- Performance tuning tips
Clang AddressSanitizer Documentation
Clang-specific guidance:
- Compilation flags and options
- Interaction with other Clang features
- Supported platforms and architectures
GCC-specific ASan documentation:
- GCC-specific flags and behavior
- Differences from Clang implementation
- Platform support in GCC
AddressSanitizer: A Fast Address Sanity Checker (USENIX Paper)
Original research paper with technical details:
- Shadow memory algorithm
- Virtual memory requirements (historically 16TB, now ~20TB)
- Performance benchmarks
- Design decisions and tradeoffs
AFL++
AFL++ is a fork of the original AFL fuzzer that offers better fuzzing performance and more advanced features while maintaining stability. A major benefit over libFuzzer is that AFL++ has stable support for running fuzzing campaigns on multiple cores, making it ideal for large-scale fuzzing efforts.
When to Use
| Fuzzer | Best For | Complexity |
|---|---|---|
| AFL++ | Multi-core fuzzing, diverse mutations, mature projects | Medium |
| libFuzzer | Quick setup, single-threaded, simple harnesses | Low |
| LibAFL | Custom fuzzers, research, advanced use cases | High |
Choose AFL++ when:
- You need multi-core fuzzing to maximize throughput
- Your project can be compiled with Clang or GCC
- You want diverse mutation strategies and mature tooling
- libFuzzer has plateaued and you need more coverage
- You're fuzzing production codebases that benefit from parallel execution
Quick Start
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
// Call your code with fuzzer-provided data
check_buf((char*)data, size);
return 0;
}
Compile and run:
# Setup AFL++ wrapper script first (see Installation)
./afl++ docker afl-clang-fast++ -DNO_MAIN=1 -O2 -fsanitize=fuzzer harness.cc main.cc -o fuzz
mkdir seeds && echo "aaaa" > seeds/minimal_seed
./afl++ docker afl-fuzz -i seeds -o out -- ./fuzz
Installation
AFL++ has many dependencies including LLVM, Python, and Rust. We recommend using a current Debian or Ubuntu distribution for fuzzing with AFL++.
| Method | When to Use | Supported Compilers |
|---|---|---|
| Ubuntu/Debian repos | Recent Ubuntu, basic features only | Ubuntu 23.10: Clang 14 & GCC 13 Debian 12: Clang 14 & GCC 12 |
| Docker (from Docker Hub) | Specific AFL++ version, Apple Silicon support | As of 4.35c: Clang 19 & GCC 11 |
| Docker (from source) | Test unreleased features, apply patches | Configurable in Dockerfile |
| From source | Avoid Docker, need specific patches | Adjustable via LLVM_CONFIG env var |
Ubuntu/Debian
Prior to installing afl++, check the clang version dependency of the packge with apt-cache show afl++, and install the matching lld version (e.g., lld-17).
apt install afl++ lld-17
Docker (from Docker Hub)
docker pull aflplusplus/aflplusplus:stable
Docker (from source)
git clone --depth 1 --branch stable https://github.com/AFLplusplus/AFLplusplus
cd AFLplusplus
docker build -t aflplusplus .
From source
Refer to the Dockerfile for Ubuntu version requirements and dependencies. Set LLVM_CONFIG to specify Clang version (e.g., llvm-config-18).
Wrapper Script Setup
Create a wrapper script to run AFL++ on host or Docker:
cat <<'EOF' > ./afl++
#!/bin/sh
AFL_VERSION="${AFL_VERSION:-"stable"}"
case "$1" in
host)
shift
bash -c "$*"
;;
docker)
shift
/usr/bin/env docker run -i \
--privileged \
-v ./:/src \
--rm \
--name "afl_fuzzing_$$" \
"aflplusplus/aflplusplus:$AFL_VERSION" \
bash -c "cd /src && bash -c \"$*\""
;;
*)
echo "Usage: $0 {host|docker}"
exit 1
;;
esac
EOF
chmod +x ./afl++
The examples below use docker mode, apart from the system configuration commands that have to reach the host kernel. Swap in host to run any of them against an AFL++ installed on the machine itself. The wrapper joins everything after the mode argument into a single shell string, so quoting does not survive: an argument containing a space (-x "my dict.dict") arrives word-split. Rename such files without spaces, or edit the wrapper for that run.
The missing -t is deliberate. docker run -ti aborts with the input device is not a TTY whenever stdin is not a terminal, which covers CI jobs and anything an agent or script drives. afl-fuzz notices there is no terminal and prints plain status lines in place of the full-screen UI. A program that insists on a terminal, such as watch, has to run on the host side of the wrapper instead. $$ expands to the wrapper's PID, so parallel instances get distinct container names rather than colliding on a single afl_fuzzing. docker ps truncates the COMMAND column, so every row looks alike; use docker ps --no-trunc to tell the instances apart before stopping one.
Security Warning: The afl-system-config and afl-persistent-config scripts require root privileges and disable OS security features. Do not fuzz on production systems or your development environment. Use a dedicated VM instead.
System Configuration
Run after each reboot for up to 15% more executions per second:
./afl++ host afl-system-config
afl-system-config tunes the kernel it runs against, so run it on the machine that hosts the campaign. ./afl++ docker afl-system-config reaches the same settings through the wrapper's --privileged container, which is the only route when AFL++ is installed via Docker alone.
For maximum performance, disable kernel security mitigations (requires grub bootloader, not supported in Docker):
./afl++ host afl-persistent-config
update-grub
reboot
./afl++ host afl-system-config
Verify with cat /proc/cmdline - output should include mitigations=off.
Writing a Harness
Harness Structure
AFL++ supports libFuzzer-style harnesses:
#include <stdint.h>
#include <stddef.h>
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
// 1. Validate input size if needed
if (size < MIN_SIZE || size > MAX_SIZE) return 0;
// 2. Call target function with fuzz data
target_function(data, size);
// 3. Return 0 (non-zero reserved for future use)
return 0;
}
Harness Rules
| Do | Don't |
|---|---|
| Reset global state between runs | Rely on state from previous runs |
| Handle edge cases gracefully | Exit on invalid input |
| Keep harness deterministic | Use random number generators |
| Free allocated memory | Create memory leaks |
| Validate input sizes | Process unbounded input |
See Also: For detailed harness writing techniques, patterns for handling complex inputs, and advanced strategies, see the fuzz-harness-writing technique skill.
Compilation
AFL++ offers multiple compilation modes with different trade-offs.
Compilation Mode Decision Tree
Choose your compilation mode:
- LTO mode (
afl-clang-lto): Best performance and instrumentation. Try this first. - LLVM mode (
afl-clang-fast): Fall back if LTO fails to compile. - GCC plugin (
afl-gcc-fast): For projects requiring GCC.
Basic Compilation (LLVM mode)
./afl++ docker afl-clang-fast++ -DNO_MAIN=1 -O2 -fsanitize=fuzzer harness.cc main.cc -o fuzz
GCC Compilation
./afl++ docker afl-g++-fast -DNO_MAIN=1 -O2 -fsanitize=fuzzer harness.cc main.cc -o fuzz
Important: GCC version must match the version used to compile the AFL++ GCC plugin.
With Sanitizers
./afl++ docker AFL_USE_ASAN=1 afl-clang-fast++ -DNO_MAIN=1 -O2 -fsanitize=fuzzer harness.cc main.cc -o fuzz
See Also: For detailed sanitizer configuration, common issues, and advanced flags, see the address-sanitizer and undefined-behavior-sanitizer technique skills.
Build Flags
Note that -g is not necessary, it is added by default by the AFL++ compilers.
| Flag | Purpose |
|---|---|
-DNO_MAIN=1 | Skip main function when using libFuzzer harness |
-O2 | Production optimization level (recommended for fuzzing) |
-fsanitize=fuzzer | Enable libFuzzer compatibility mode and adds the fuzzer runtime when linking executable |
-fsanitize=fuzzer-no-link | Instrument without linking fuzzer runtime (for static libraries and object files) |
Corpus Management
Creating Initial Corpus
AFL++ requires at least one non-empty seed file:
mkdir seeds
echo "aaaa" > seeds/minimal_seed
For real projects, gather representative inputs:
- Download example files for the format you're fuzzing
- Extract test cases from the project's test suite
- Use minimal valid inputs for your file format
Corpus Minimization
After a campaign, minimize the corpus to keep only unique coverage:
./afl++ docker afl-cmin -i out/default/queue -o minimized_corpus -- ./fuzz
See Also: For corpus creation strategies, dictionaries, and seed selection, see the fuzzing-corpus technique skill.
Running Campaigns
Basic Run
./afl++ docker afl-fuzz -i seeds -o out -- ./fuzz
Setting Environment Variables
./afl++ docker AFL_FAST_CAL=1 afl-fuzz -i seeds -o out -- ./fuzz
Interpreting Output
AFL++ reports these statistics either way, but how you read them depends on the
mode. The wrapper's docker run -i gives the container no TTY, so afl-fuzz
drops the full-screen UI and writes plain status lines to the log instead — the
fields below appear there, and in state/<instance>/fuzzer_stats. To get the
interactive UI, run host mode in a terminal, or add -t to the wrapper for a
run you are watching by hand.
| Output | Meaning |
|---|---|
| execs/sec | Execution speed - higher is better |
| cycles done | Number of queue passes completed |
| corpus count | Number of unique test cases in queue |
| saved crashes | Number of unique crashes found |
| stability | % of stable edges (should be near 100%) |
Output Directory Structure
out/default/
├── cmdline # How was the SUT invoked?
├── crashes/ # Inputs that crash the SUT
│ └── id:000000,sig:06,src:000002,time:286,execs:13105,op:havoc,rep:4
├── hangs/ # Inputs that hang the SUT
├── queue/ # Test cases reproducing final fuzzer state
│ ├── id:000000,time:0,execs:0,orig:minimal_seed
│ └── id:000001,src:000000,time:0,execs:8,op:havoc,rep:6,+cov
├── fuzzer_stats # Campaign statistics
└── plot_data # Data for plotting
Analyzing Results
View live campaign statistics:
./afl++ docker afl-whatsup out
Create coverage plots. The aflplusplus image already ships gnuplot-nox; in host mode, install gnuplot first with apt install gnuplot.
./afl++ docker afl-plot out/default out_graph/
Re-executing Test Cases
Pass one of the filenames from out/default/crashes/:
./afl++ docker ./fuzz out/default/crashes/id:000000,sig:06,src:000002,time:286,execs:13105,op:havoc,rep:4
Fuzzer Options
| Option | Purpose |
|---|---|
-G 4000 | Maximum test input length (default: 1048576 bytes) |
-t 1000 | Timeout in milliseconds for each test case (default: 1000ms) |
-m 1000 | Memory limit in megabytes (default: 0 = unlimited) |
-x ./dict.dict | Use dictionary file to guide mutations |
Environment Variables That Matter
AFL++ has many environment variables, but most are niche. These are the ones that matter in practice.
Always Set These
# Every campaign should use tmpfs — SSDs will thank you, and it's faster
AFL_TMPDIR=/dev/shm
AFL_TMPDIR is a free performance win with no downsides — not setting it wears out your SSD and slows fuzzing.
Slow Targets
# Speeds up calibration ~2.5x — use when targets are slow (e.g., >10 ms/exec)
AFL_FAST_CAL=1
AFL_FAST_CAL reduces calibration time with negligible precision loss. Recommended specifically for slow targets where calibration would otherwise take a long time.
Multi-Core Campaigns
# On the primary (-M) instance only — needed for afl-cmin, not for fuzzing itself
AFL_FINAL_SYNC=1
# On all instances — cache test cases in memory (default: 50 MB, good range: 50-250 MB)
AFL_TESTCACHE_SIZE=100
AFL_FINAL_SYNC tells the primary instance to do a final import from all secondaries when stopping. This does not affect the fuzzing process itself — it only matters when you later run afl-cmin for corpus minimization, ensuring the primary's queue has the full combined corpus. AFL_TESTCACHE_SIZE caches test cases in memory to reduce disk I/O; the default is 50 MB and values between 50-250 MB work well for most campaigns.
CI/Automated Fuzzing
# Fail fast if fuzzing isn't finding anything
AFL_EXIT_ON_TIME=3600 # 1 hour with no new paths = stop
# Or run until "done" (all queue entries processed)
AFL_EXIT_WHEN_DONE=1
# Headless environments
AFL_NO_UI=1
Unbounded fuzzing in CI wastes resources. Set time limits or use exit conditions.
Variables to Avoid
| Variable | Why Skip It |
|---|---|
AFL_NO_ARITH | Can hurt coverage on binary formats, but may be useful for text-based targets |
AFL_SHUFFLE_QUEUE | Only for exotic setups, usually harmful |
AFL_DISABLE_TRIM | Trimming is valuable, don't disable without reason |
Multi-Core Fuzzing
AFL++ excels at multi-core fuzzing with two major advantages:
- More executions per second (scales linearly with physical cores)
- Asymmetrical fuzzing (e.g., one ASan job, rest without sanitizers)
Starting a Campaign
Start the primary fuzzer (in background):
./afl++ docker afl-fuzz -M primary -i seeds -o state -- ./fuzz 1>primary.log 2>primary.error </dev/null &
Start secondary fuzzers (as many as you have cores):
./afl++ docker afl-fuzz -S secondary01 -i seeds -o state -- ./fuzz 1>secondary01.log 2>secondary01.error </dev/null &
./afl++ docker afl-fuzz -S secondary02 -i seeds -o state -- ./fuzz 1>secondary02.log 2>secondary02.error </dev/null &
The </dev/null is required, not decorative. docker run -i keeps the client
reading its own stdin, and a backgrounded process that reads the terminal is sent
SIGTTIN, whose default action stops it — so without the redirect these jobs show
up as Stopped in jobs and never fuzz.
Monitoring Multi-Core Campaigns
List all running jobs:
jobs
View live statistics. watch needs a terminal that docker run -i does not give it, so wrap the whole invocation instead of running watch inside the container. Every tick starts a container, which is why the interval is 5 seconds rather than 1:
watch -n5 --color ./afl++ docker afl-whatsup state/
Stopping All Fuzzers
kill $(jobs -p)
Coverage Analysis
AFL++ automatically tracks coverage through edge instrumentation. Coverage information is stored in fuzzer_stats and plot_data.
Measuring Coverage
Use afl-plot to visualize coverage over time:
./afl++ docker afl-plot out/default out_graph/
Improving Coverage
- Use dictionaries for format-aware fuzzing
- Run longer campaigns (cycles_wo_finds indicates plateau)
- Try different mutation strategies with multi-core fuzzing
- Analyze coverage gaps and add targeted seed inputs
See Also: For detailed coverage analysis techniques, identifying coverage gaps, and systematic coverage improvement, see the coverage-analysis technique skill.
CMPLOG
CMPLOG/RedQueen is the best path constraint solving mechanism available in any fuzzer. To enable it, the fuzz target needs to be instrumented for it. Before building the fuzzing target set the environment variable:
./afl++ docker AFL_LLVM_CMPLOG=1 make
No special action is needed for compiling and linking the harness.
To run a fuzzer instance with a CMPLOG instrumented fuzzing target, add -c0 to the command like arguments:
./afl++ docker afl-fuzz -c0 -S cmplog -i seeds -o state -- ./fuzz 1>cmplog.log 2>cmplog.error </dev/null &
Sanitizer Integration
Sanitizers are essential for finding memory corruption bugs that don't cause immediate crashes.
AddressSanitizer (ASan)
./afl++ docker AFL_USE_ASAN=1 afl-clang-fast++ -DNO_MAIN=1 -O2 -fsanitize=fuzzer harness.cc main.cc -o fuzz
Note: Memory limit (-m) is not supported with ASan due to 20TB virtual memory reservation.
UndefinedBehaviorSanitizer (UBSan)
./afl++ docker AFL_USE_UBSAN=1 afl-clang-fast++ -DNO_MAIN=1 -O2 -fsanitize=fuzzer,undefined harness.cc main.cc -o fuzz
Common Sanitizer Issues
| Issue | Solution |
|---|---|
| ASan slows fuzzing | Use only 1 ASan job in multi-core setup |
| Stack exhaustion | Increase stack with ASAN_OPTIONS=stack_size=... |
| GCC version mismatch | Ensure system GCC matches AFL++ plugin version |
See Also: For comprehensive sanitizer configuration and troubleshooting, see the address-sanitizer technique skill.
Advanced Usage
Tips and Tricks
| Tip | Why It Helps |
|---|---|
| Use LLVMFuzzerTestOneInput harnesses where possible | If a fuzzing campaign has at least 85% stability then this is the most efficient fuzzing style. If not then try standard input or file input fuzzing |
| Use dictionaries | Helps fuzzer discover format-specific keywords and magic bytes |
| Set realistic timeouts | Prevents false positives from system load |
| Limit input size | Larger inputs don't necessarily explore more space |
| Monitor stability | Low stability indicates non-deterministic behavior |
Standard Input Fuzzing
AFL++ can fuzz programs reading from stdin without a libFuzzer harness:
./afl++ docker afl-clang-fast++ -O2 main_stdin.c -o fuzz_stdin
./afl++ docker afl-fuzz -i seeds -o out -- ./fuzz_stdin
This is slower than persistent mode but requires no harness code.
File Input Fuzzing
For programs that read files, use @@ placeholder:
./afl++ docker afl-clang-fast++ -O2 main_file.c -o fuzz_file
./afl++ docker afl-fuzz -i seeds -o out -- ./fuzz_file @@
For better performance, use fmemopen to create file descriptors from memory.
Argument Fuzzing
Fuzz command-line arguments using argv-fuzz-inl.h:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#ifdef __AFL_COMPILER
#include "argv-fuzz-inl.h"
#endif
void check_buf(char *buf, size_t buf_len) {
if(buf_len > 0 && buf[0] == 'a') {
if(buf_len > 1 && buf[1] == 'b') {
if(buf_len > 2 && buf[2] == 'c') {
abort();
}
}
}
}
int main(int argc, char *argv[]) {
#ifdef __AFL_COMPILER
AFL_INIT_ARGV();
#endif
if (argc < 2) {
fprintf(stderr, "Usage: %s <input_string>\n", argv[0]);
return 1;
}
char *input_buf = argv[1];
size_t len = strlen(input_buf);
check_buf(input_buf, len);
return 0;
}
Download the header:
curl -O https://raw.githubusercontent.com/AFLplusplus/AFLplusplus/stable/utils/argv_fuzzing/argv-fuzz-inl.h
Compile and run:
./afl++ docker afl-clang-fast++ -O2 main_arg.c -o fuzz_arg
./afl++ docker afl-fuzz -i seeds -o out -- ./fuzz_arg
Performance Tuning
| Setting | Impact |
|---|---|
| CPU core count | Linear scaling with physical cores |
| Persistent mode | 10-20x faster than fork server |
-G input size limit | Smaller = faster, but may miss bugs |
| ASan ratio | 1 ASan job per 4-8 non-ASan jobs |
Troubleshooting
| Problem | Cause | Solution |
|---|---|---|
| Low exec/sec (<1k) | Not using persistent mode | Create a LLVMFuzzerTestOneInput style harness |
| Low stability (<85%) | Non-deterministic code | Fuzz a program via stdin or file inputs, or create such a harness |
| GCC plugin error | GCC version mismatch | Ensure system GCC matches AFL++ build and install gcc-$GCC_VERSION-plugin-dev |
| No crashes found | Need sanitizers | Recompile with AFL_USE_ASAN=1 |
| Memory limit exceeded | ASan uses 20TB virtual | Remove -m flag when using ASan |
| Docker performance loss | Virtualization overhead | Use bare metal or VM for production fuzzing |
Related Skills
Technique Skills
| Skill | Use Case |
|---|---|
| fuzz-harness-writing | Detailed guidance on writing effective harnesses |
| address-sanitizer | Memory error detection during fuzzing |
| undefined-behavior-sanitizer | Detect undefined behavior bugs |
| fuzzing-corpus | Building and managing seed corpora |
| fuzzing-dictionaries | Creating dictionaries for format-aware fuzzing |
Related Fuzzers
| Skill | When to Consider |
|---|---|
| libfuzzer | Quick prototyping, single-threaded fuzzing is sufficient |
| libafl | Need custom mutators or research-grade features |
Resources
Key External Resources
AFL++ GitHub Repository Official repository with comprehensive documentation, examples, and issue tracker.
Fuzzing in Depth Advanced documentation by the AFL++ team covering instrumentation modes, optimization techniques, and advanced use cases.
AFL++ Under The Hood Technical deep-dive into AFL++ internals, mutation strategies, and coverage tracking mechanisms.
AFL++: Combining Incremental Steps of Fuzzing Research Research paper describing AFL++ architecture and performance improvements over original AFL.
Video Resources
- Fuzzing cURL - Trail of Bits blog post on using AFL++ argument fuzzing for cURL
- Sudo Vulnerability Walkthrough - LiveOverflow series on rediscovering CVE-2021-3156
- Rediscovery of libpng bug - LiveOverflow video on finding CVE-2023-4863
Atheris
Atheris is a coverage-guided Python fuzzer built on libFuzzer. It enables fuzzing of both pure Python code and Python C extensions with integrated AddressSanitizer support for detecting memory corruption issues.
When to Use
| Fuzzer | Best For | Complexity |
|---|---|---|
| Atheris | Python code and C extensions | Low-Medium |
| Hypothesis | Property-based testing | Low |
| python-afl | AFL-style fuzzing | Medium |
Choose Atheris when:
- Fuzzing pure Python code with coverage guidance
- Testing Python C extensions for memory corruption
- Integration with libFuzzer ecosystem is desired
- AddressSanitizer support is needed
Quick Start
import sys
import atheris
@atheris.instrument_func
def TestOneInput(data: bytes):
if len(data) == 4:
if data[0] == 0x46: # "F"
if data[1] == 0x55: # "U"
if data[2] == 0x5A: # "Z"
if data[3] == 0x5A: # "Z"
raise RuntimeError("You caught me")
def main():
atheris.Setup(sys.argv, TestOneInput)
atheris.Fuzz()
if __name__ == "__main__":
main()
Run:
uv run python fuzz.py
Installation
Atheris supports 32-bit and 64-bit Linux, and macOS. We recommend fuzzing on Linux because it's simpler to manage and often faster.
Prerequisites
- Python 3.7 or later
- Recent version of clang (preferably latest release)
- For Docker users: Docker Desktop
Linux/macOS
uv init --bare # once, if the harness directory is not yet a uv project
uv add atheris
Docker Environment (Recommended)
For a fully operational Linux environment with all dependencies configured:
# https://hub.docker.com/_/python
ARG PYTHON_VERSION=3.11
FROM python:$PYTHON_VERSION-slim-bookworm
RUN python --version
RUN apt update && apt install -y \
ca-certificates \
wget \
&& rm -rf /var/lib/apt/lists/*
# LLVM builds version 15-19 for Debian 12 (Bookworm)
# https://apt.llvm.org/bookworm/dists/
ARG LLVM_VERSION=19
RUN echo "deb http://apt.llvm.org/bookworm/ llvm-toolchain-bookworm-$LLVM_VERSION main" > /etc/apt/sources.list.d/llvm.list
RUN echo "deb-src http://apt.llvm.org/bookworm/ llvm-toolchain-bookworm-$LLVM_VERSION main" >> /etc/apt/sources.list.d/llvm.list
RUN wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key > /etc/apt/trusted.gpg.d/apt.llvm.org.asc
RUN apt update && apt install -y \
build-essential \
clang-$LLVM_VERSION \
&& rm -rf /var/lib/apt/lists/*
ENV APP_DIR "/app"
RUN mkdir $APP_DIR
WORKDIR $APP_DIR
ENV VIRTUAL_ENV "/opt/venv"
RUN python -m venv $VIRTUAL_ENV
ENV PATH "$VIRTUAL_ENV/bin:$PATH"
# https://github.com/google/atheris/blob/master/native_extension_fuzzing.md#step-1-compiling-your-extension
ENV CC="clang-$LLVM_VERSION"
ENV CFLAGS "-fsanitize=address,fuzzer-no-link"
ENV CXX="clang++-$LLVM_VERSION"
ENV CXXFLAGS "-fsanitize=address,fuzzer-no-link"
ENV LDSHARED="clang-$LLVM_VERSION -shared"
ENV LDSHAREDXX="clang++-$LLVM_VERSION -shared"
ENV ASAN_SYMBOLIZER_PATH="/usr/bin/llvm-symbolizer-$LLVM_VERSION"
# Allow Atheris to find fuzzer sanitizer shared libs
# https://github.com/google/atheris#building-from-source
RUN LIBFUZZER_LIB=$($CC -print-file-name=libclang_rt.fuzzer_no_main-$(uname -m).a) \
python -m pip install --no-binary atheris atheris
# https://github.com/google/atheris/blob/master/native_extension_fuzzing.md#option-a-sanitizerlibfuzzer-preloads
ENV LD_PRELOAD "$VIRTUAL_ENV/lib/python3.11/site-packages/asan_with_fuzzer.so"
# 1. Skip memory allocation failures for now, they are common, and low impact (DoS)
# 2. https://github.com/google/atheris/blob/master/native_extension_fuzzing.md#leak-detection
ENV ASAN_OPTIONS "allocator_may_return_null=1,detect_leaks=0"
CMD ["/bin/bash"]
Build and run:
docker build -t atheris .
docker run -it atheris
Verification
python -c "import atheris; print(atheris.__version__)"
Writing a Harness
Harness Structure for Pure Python
import sys
import atheris
@atheris.instrument_func
def TestOneInput(data: bytes):
"""
Fuzzing entry point. Called with random byte sequences.
Args:
data: Random bytes generated by the fuzzer
"""
# Add input validation if needed
if len(data) < 1:
return
# Call your target function
try:
your_target_function(data)
except ValueError:
# Expected exceptions should be caught
pass
# Let unexpected exceptions crash (that's what we're looking for!)
def main():
atheris.Setup(sys.argv, TestOneInput)
atheris.Fuzz()
if __name__ == "__main__":
main()
Structured Input with FuzzedDataProvider
A target taking several typed arguments wastes most of the fuzzer's inputs if the harness
slices data by hand, because every mutation shifts the byte offsets of everything after it.
atheris.FuzzedDataProvider splits one bytes input into typed values instead:
fdp = atheris.FuzzedDataProvider(data)
name = fdp.ConsumeUnicodeNoSurrogates(fdp.ConsumeIntInRange(0, 64))
strict = fdp.ConsumeBool()
See structured-input.md for the full method reference, the fixed-draw- order rule, and what each method returns once the buffer runs dry.
Harness Rules
| Do | Don't |
|---|---|
Use @atheris.instrument_func for coverage | Forget to instrument target code |
| Catch expected exceptions | Catch all exceptions indiscriminately |
Use atheris.instrument_imports() for libraries | Import modules after atheris.Setup() |
| Keep harness deterministic | Use randomness or time-based behavior |
See Also: For detailed harness writing techniques, patterns for handling complex inputs, and advanced strategies, see the fuzz-harness-writing technique skill.
Fuzzing Pure Python Code
For fuzzing broader parts of an application or library, use instrumentation functions:
import atheris
with atheris.instrument_imports():
import your_module
from another_module import target_function
def TestOneInput(data: bytes):
target_function(data)
atheris.Setup(sys.argv, TestOneInput)
atheris.Fuzz()
Instrumentation Options:
atheris.instrument_func- Decorator for single function instrumentationatheris.instrument_imports()- Context manager for instrumenting all imported modulesatheris.instrument_all()- Instrument all Python code system-wide
Fuzzing Python C Extensions
Python C extensions require compilation with specific flags for instrumentation and sanitizer support.
Environment Configuration
If using the provided Dockerfile, these are already configured. For local setup:
export CC="clang"
export CFLAGS="-fsanitize=address,fuzzer-no-link"
export CXX="clang++"
export CXXFLAGS="-fsanitize=address,fuzzer-no-link"
export LDSHARED="clang -shared"
Example: Fuzzing cbor2
Install the extension from source:
CBOR2_BUILD_C_EXTENSION=1 uv add --no-binary-package cbor2 'cbor2==5.6.4'
The --no-binary-package flag ensures the C extension is compiled locally with
instrumentation rather than pulled as a prebuilt wheel. Persist that choice with
no-binary-package = ["cbor2"] under [tool.uv] in pyproject.toml, or a later
uv sync can silently swap in an uninstrumented wheel.
Create cbor2-fuzz.py:
import sys
import atheris
# _cbor2 ensures the C library is imported
from _cbor2 import loads
def TestOneInput(data: bytes):
try:
loads(data)
except Exception:
# We're searching for memory corruption, not Python exceptions
pass
def main():
atheris.Setup(sys.argv, TestOneInput)
atheris.Fuzz()
if __name__ == "__main__":
main()
Run:
uv run python cbor2-fuzz.py
Important: When running locally (not in Docker), you must set
LD_PRELOADmanually.
Corpus Management
Creating Initial Corpus
mkdir corpus
# Add seed inputs
echo "test data" > corpus/seed1
echo '{"key": "value"}' > corpus/seed2
Run with corpus:
uv run python fuzz.py corpus/
Corpus Minimization
Atheris inherits corpus minimization from libFuzzer:
uv run python fuzz.py -merge=1 new_corpus/ old_corpus/
See Also: For corpus creation strategies, dictionaries, and seed selection, see the fuzzing-corpus technique skill.
Running Campaigns
Basic Run
uv run python fuzz.py
With Corpus Directory
uv run python fuzz.py corpus/
Common Options
# Run for 10 minutes
uv run python fuzz.py -max_total_time=600
# Limit input size
uv run python fuzz.py -max_len=1024
# Run with multiple workers
uv run python fuzz.py -workers=4 -jobs=4
Interpreting Output
| Output | Meaning |
|---|---|
NEW cov: X | Found new coverage, corpus expanded |
pulse cov: X | Periodic status update |
exec/s: X | Executions per second (throughput) |
corp: X/Yb | Corpus size: X inputs, Y bytes total |
ERROR: libFuzzer | Crash detected |
Sanitizer Integration
AddressSanitizer (ASan)
AddressSanitizer is automatically integrated when using the provided Docker environment or when compiling with appropriate flags.
For local setup:
export CFLAGS="-fsanitize=address,fuzzer-no-link"
export CXXFLAGS="-fsanitize=address,fuzzer-no-link"
Configure ASan behavior:
export ASAN_OPTIONS="allocator_may_return_null=1,detect_leaks=0"
LD_PRELOAD Configuration
For native extension fuzzing:
export LD_PRELOAD="$(python -c 'import atheris; import os; print(os.path.join(os.path.dirname(atheris.__file__), "asan_with_fuzzer.so"))')"
See Also: For detailed sanitizer configuration, common issues, and advanced flags, see the address-sanitizer and undefined-behavior-sanitizer technique skills.
Common Sanitizer Issues
| Issue | Solution |
|---|---|
LD_PRELOAD not set | Export LD_PRELOAD to point to asan_with_fuzzer.so |
| Memory allocation failures | Set ASAN_OPTIONS=allocator_may_return_null=1 |
| Leak detection noise | Set ASAN_OPTIONS=detect_leaks=0 |
| Missing symbolizer | Set ASAN_SYMBOLIZER_PATH to llvm-symbolizer |
Advanced Usage
Tips and Tricks
| Tip | Why It Helps |
|---|---|
Use atheris.instrument_imports() early | Ensures all imports are instrumented for coverage |
Start with small max_len | Faster initial fuzzing, gradually increase |
| Use dictionaries for structured formats | Helps fuzzer understand format tokens |
| Run multiple parallel instances | Better coverage exploration |
Custom Instrumentation
Fine-tune what gets instrumented:
import atheris
# Instrument only specific modules
with atheris.instrument_imports():
import target_module
# Don't instrument test harness code
def TestOneInput(data: bytes):
target_module.parse(data)
Performance Tuning
| Setting | Impact |
|---|---|
-max_len=N | Smaller values = faster execution |
-workers=N -jobs=N | Parallel fuzzing for faster coverage |
ASAN_OPTIONS=fast_unwind_on_malloc=0 | Better stack traces, slower execution |
UndefinedBehaviorSanitizer (UBSan)
Add UBSan to catch additional bugs:
export CFLAGS="-fsanitize=address,undefined,fuzzer-no-link"
export CXXFLAGS="-fsanitize=address,undefined,fuzzer-no-link"
Note: Modify flags in Dockerfile if using containerized setup.
Real-World Examples
Two complete harnesses — a pure-Python parser and an HTTP response parser — are in examples.md.
Troubleshooting
| Problem | Cause | Solution |
|---|---|---|
| No coverage increase | Poor seed corpus or target not instrumented | Add better seeds, verify instrument_imports() |
| Slow execution | ASan overhead or large inputs | Reduce max_len, use ASAN_OPTIONS=fast_unwind_on_malloc=1 |
| Import errors | Modules imported before instrumentation | Move imports inside instrument_imports() context |
| Segfault without ASan output | Missing LD_PRELOAD | Set LD_PRELOAD to asan_with_fuzzer.so path |
| Build failures | Wrong compiler or missing flags | Verify CC, CFLAGS, and clang version |
Related Skills
Technique Skills
| Skill | Use Case |
|---|---|
| fuzz-harness-writing | Detailed guidance on writing effective harnesses |
| address-sanitizer | Memory error detection during fuzzing |
| undefined-behavior-sanitizer | Catching undefined behavior in C extensions |
| coverage-analysis | Measuring and improving code coverage |
| fuzzing-corpus | Building and managing seed corpora |
Related Fuzzers
| Skill | When to Consider |
|---|---|
| hypothesis | Property-based testing with type-aware generation |
| python-afl | AFL-style fuzzing for Python when Atheris isn't available |
Resources
Key External Resources
Atheris GitHub Repository Official repository with installation instructions, examples, and documentation for fuzzing both pure Python and native extensions.
Native Extension Fuzzing Guide Comprehensive guide covering compilation flags, LD_PRELOAD setup, sanitizer configuration, and troubleshooting for Python C extensions.
Continuously Fuzzing Python C Extensions Trail of Bits blog post covering CI/CD integration, ClusterFuzzLite setup, and real-world examples of fuzzing Python C extensions in continuous integration pipelines.
ClusterFuzzLite Python Integration Guide for integrating Atheris fuzzing into CI/CD pipelines using ClusterFuzzLite for automated continuous fuzzing.
Video Resources
Videos and tutorials are available in the main Atheris documentation and libFuzzer resources.
cargo-fuzz
cargo-fuzz is the de facto choice for fuzzing Rust projects when using Cargo. It uses libFuzzer as the backend and provides a convenient Cargo subcommand that automatically enables relevant compilation flags for your Rust project, including support for sanitizers like AddressSanitizer.
When to Use
cargo-fuzz is currently the primary and most mature fuzzing solution for Rust projects using Cargo.
| Fuzzer | Best For | Complexity |
|---|---|---|
| cargo-fuzz | Cargo-based Rust projects, quick setup | Low |
| AFL++ | Multi-core fuzzing, non-Cargo projects | Medium |
| LibAFL | Custom fuzzers, research, advanced use cases | High |
Choose cargo-fuzz when:
- Your project uses Cargo (required)
- You want simple, quick setup with minimal configuration
- You need integrated sanitizer support
- You're fuzzing Rust code with or without unsafe blocks
Quick Start
#![no_main]
use libfuzzer_sys::fuzz_target;
fn harness(data: &[u8]) {
your_project::check_buf(data);
}
fuzz_target!(|data: &[u8]| {
harness(data);
});
Initialize and run:
cargo fuzz init
# Edit fuzz/fuzz_targets/fuzz_target_1.rs with your harness
cargo +nightly fuzz run fuzz_target_1
Installation
cargo-fuzz requires the nightly Rust toolchain because it uses features only available in nightly.
Prerequisites
- Rust and Cargo installed via rustup
- Nightly toolchain
Linux/macOS
# Install nightly toolchain
rustup install nightly
# Install cargo-fuzz
cargo install cargo-fuzz
Verification
cargo +nightly --version
cargo fuzz --version
Writing a Harness
Project Structure
cargo-fuzz works best when your code is structured as a library crate. If you have a binary project, split your main.rs into:
src/main.rs # Entry point (main function)
src/lib.rs # Code to fuzz (public functions)
Cargo.toml
Initialize fuzzing:
cargo fuzz init
This creates:
fuzz/
├── Cargo.toml
└── fuzz_targets/
└── fuzz_target_1.rs
Harness Structure
#![no_main]
use libfuzzer_sys::fuzz_target;
fn harness(data: &[u8]) {
// 1. Validate input size if needed
if data.is_empty() {
return;
}
// 2. Call target function with fuzz data
your_project::target_function(data);
}
fuzz_target!(|data: &[u8]| {
harness(data);
});
Harness Rules
| Do | Don't |
|---|---|
| Structure code as library crate | Keep everything in main.rs |
Use fuzz_target! macro | Write custom main function |
Handle Result::Err gracefully | Panic on expected errors |
| Keep harness deterministic | Use random number generators |
See Also: For detailed harness writing techniques and structure-aware fuzzing with the
arbitrarycrate, see the fuzz-harness-writing technique skill.
Structure-Aware Fuzzing
cargo-fuzz integrates with the arbitrary crate for structure-aware fuzzing:
// In your library crate
use arbitrary::Arbitrary;
#[derive(Debug, Arbitrary)]
pub struct Name {
data: String
}
// In your fuzz target
#![no_main]
use libfuzzer_sys::fuzz_target;
fuzz_target!(|data: your_project::Name| {
data.check_buf();
});
Add to your library's Cargo.toml:
[dependencies]
arbitrary = { version = "1", features = ["derive"] }
Running Campaigns
Basic Run
cargo +nightly fuzz run fuzz_target_1
Without Sanitizers (Safe Rust)
If your project doesn't use unsafe Rust, disable sanitizers for 2x performance boost:
cargo +nightly fuzz run --sanitizer none fuzz_target_1
Check if your project uses unsafe code:
cargo install cargo-geiger
cargo geiger
Re-executing Test Cases
# Run a specific test case (e.g., a crash)
cargo +nightly fuzz run fuzz_target_1 fuzz/artifacts/fuzz_target_1/crash-<hash>
# Run all corpus entries without fuzzing
cargo +nightly fuzz run fuzz_target_1 fuzz/corpus/fuzz_target_1 -- -runs=0
Using Dictionaries
cargo +nightly fuzz run fuzz_target_1 -- -dict=./dict.dict
Interpreting Output
| Output | Meaning |
|---|---|
NEW | New coverage-increasing input discovered |
pulse | Periodic status update |
INITED | Fuzzer initialized successfully |
| Crash with stack trace | Bug found, saved to fuzz/artifacts/ |
Corpus location: fuzz/corpus/fuzz_target_1/
Crashes location: fuzz/artifacts/fuzz_target_1/
Sanitizer Integration
AddressSanitizer (ASan)
ASan is enabled by default and detects memory errors:
cargo +nightly fuzz run fuzz_target_1
Disabling Sanitizers
For pure safe Rust (no unsafe blocks in your code or dependencies):
cargo +nightly fuzz run --sanitizer none fuzz_target_1
Performance impact: ASan adds ~2x overhead. Disable for safe Rust to improve fuzzing speed.
Checking for Unsafe Code
cargo install cargo-geiger
cargo geiger
See Also: For detailed sanitizer configuration, flags, and troubleshooting, see the address-sanitizer technique skill.
Coverage Analysis
cargo-fuzz integrates with Rust's coverage tools to analyze fuzzing effectiveness.
Prerequisites
rustup toolchain install nightly --component llvm-tools-preview
cargo install cargo-binutils
cargo install rustfilt
Generating Coverage Reports
# Generate coverage data from corpus
cargo +nightly fuzz coverage fuzz_target_1
Create coverage generation script:
cat <<'EOF' > ./generate_html
#!/bin/sh
if [ $# -lt 1 ]; then
echo "Error: Name of fuzz target is required."
echo "Usage: $0 fuzz_target [sources...]"
exit 1
fi
FUZZ_TARGET="$1"
shift
SRC_FILTER="$@"
TARGET=$(rustc -vV | sed -n 's|host: ||p')
cargo +nightly cov -- show -Xdemangler=rustfilt \
"target/$TARGET/coverage/$TARGET/release/$FUZZ_TARGET" \
-instr-profile="fuzz/coverage/$FUZZ_TARGET/coverage.profdata" \
-show-line-counts-or-regions -show-instantiations \
-format=html -o fuzz_html/ $SRC_FILTER
EOF
chmod +x ./generate_html
Generate HTML report:
./generate_html fuzz_target_1 src/lib.rs
HTML report saved to: fuzz_html/
See Also: For detailed coverage analysis techniques and systematic coverage improvement, see the coverage-analysis technique skill.
Advanced Usage
Tips and Tricks
| Tip | Why It Helps |
|---|---|
| Start with a seed corpus | Dramatically speeds up initial coverage discovery |
Use --sanitizer none for safe Rust | 2x performance improvement |
| Check coverage regularly | Identifies gaps in harness or seed corpus |
| Use dictionaries for parsers | Helps overcome magic value checks |
| Structure code as library | Required for cargo-fuzz integration |
libFuzzer Options
Pass options to libFuzzer after --:
# See all options
cargo +nightly fuzz run fuzz_target_1 -- -help=1
# Set timeout per run
cargo +nightly fuzz run fuzz_target_1 -- -timeout=10
# Use dictionary
cargo +nightly fuzz run fuzz_target_1 -- -dict=dict.dict
# Limit maximum input size
cargo +nightly fuzz run fuzz_target_1 -- -max_len=1024
Multi-Core Fuzzing
# Experimental forking support (not recommended)
cargo +nightly fuzz run --jobs 1 fuzz_target_1
Note: The multi-core fuzzing feature is experimental and not recommended. For parallel fuzzing, consider running multiple instances manually or using AFL++.
Real-World Examples
Example: ogg Crate
The ogg crate parses Ogg media container files. Parsers are excellent fuzzing targets because they handle untrusted data.
# Clone and initialize
git clone https://github.com/RustAudio/ogg.git
cd ogg/
cargo fuzz init
Harness at fuzz/fuzz_targets/fuzz_target_1.rs:
#![no_main]
use ogg::{PacketReader, PacketWriter};
use ogg::writing::PacketWriteEndInfo;
use std::io::Cursor;
use libfuzzer_sys::fuzz_target;
fn harness(data: &[u8]) {
let mut pck_rdr = PacketReader::new(Cursor::new(data.to_vec()));
pck_rdr.delete_unread_packets();
let output = Vec::new();
let mut pck_wtr = PacketWriter::new(Cursor::new(output));
if let Ok(_) = pck_rdr.read_packet() {
if let Ok(r) = pck_rdr.read_packet() {
match r {
Some(pck) => {
let inf = if pck.last_in_stream() {
PacketWriteEndInfo::EndStream
} else if pck.last_in_page() {
PacketWriteEndInfo::EndPage
} else {
PacketWriteEndInfo::NormalPacket
};
let stream_serial = pck.stream_serial();
let absgp_page = pck.absgp_page();
let _ = pck_wtr.write_packet(
pck.data, stream_serial, inf, absgp_page
);
}
None => return,
}
}
}
}
fuzz_target!(|data: &[u8]| {
harness(data);
});
Seed the corpus:
mkdir fuzz/corpus/fuzz_target_1/
curl -o fuzz/corpus/fuzz_target_1/320x240.ogg \
https://commons.wikimedia.org/wiki/File:320x240.ogg
Run:
cargo +nightly fuzz run fuzz_target_1
Analyze coverage:
cargo +nightly fuzz coverage fuzz_target_1
./generate_html fuzz_target_1 src/lib.rs
Troubleshooting
| Problem | Cause | Solution |
|---|---|---|
| "requires nightly" error | Using stable toolchain | Use cargo +nightly fuzz |
| Slow fuzzing performance | ASan enabled for safe Rust | Add --sanitizer none flag |
| "cannot find binary" | No library crate | Move code from main.rs to lib.rs |
| Sanitizer compilation issues | Wrong nightly version | Try different nightly: rustup install nightly-2024-01-01 |
| Low coverage | Missing seed corpus | Add sample inputs to fuzz/corpus/fuzz_target_1/ |
| Magic value not found | No dictionary | Create dictionary file with magic values |
Related Skills
Technique Skills
| Skill | Use Case |
|---|---|
| fuzz-harness-writing | Structure-aware fuzzing with arbitrary crate |
| address-sanitizer | Understanding ASan output and configuration |
| coverage-analysis | Measuring and improving fuzzing effectiveness |
| fuzzing-corpus | Building and managing seed corpora |
| fuzzing-dictionaries | Creating dictionaries for format-aware fuzzing |
Related Fuzzers
| Skill | When to Consider |
|---|---|
| libfuzzer | Fuzzing C/C++ code with similar workflow |
| aflpp | Multi-core fuzzing or non-Cargo Rust projects |
| libafl | Advanced fuzzing research or custom fuzzer development |
Resources
Rust Fuzz Book - cargo-fuzz Official documentation for cargo-fuzz covering installation, usage, and advanced features.
arbitrary crate documentation Guide to structure-aware fuzzing with automatic derivation for Rust types.
cargo-fuzz GitHub Repository Source code, issue tracker, and examples for cargo-fuzz.
Constant-Time Analysis
Compile the code, inspect the emitted assembly or bytecode for variable-time instructions, then decide which of the flagged operations actually touch secrets. The compilation step is mechanical; the triage step is the work.
When to Use
- Implementing or reviewing a signature, encryption, KEM, or key derivation routine
- Code applies
/or%to a value derived from a key, plaintext, nonce, or token - The user mentions "constant-time", "timing attack", "side-channel", or "KyberSlash"
- Reviewing functions named
sign,verify,encrypt,decrypt,derive_key
When NOT to Use
- Measuring timing variance on a running binary — use the
constant-time-testingskill from thetesting-handbook-skillsplugin, which covers dudect and statistical approaches and may not be installed. This skill inspects compiler output statically and never executes the code under test. - Non-cryptographic code, or crypto code where every input is public
- High-level API usage where a vetted library owns the constant-time guarantees
- Cache and other microarchitectural side channels — the assembly view cannot see them
Language Routing
Read the guide for the target language before interpreting any findings; each one lists that language's dangerous instructions and the idiomatic constant-time replacements.
| Guide | Languages |
|---|---|
| references/compiled.md | C, C++, Go, Rust |
| references/swift.md | Swift |
| references/vm-compiled.md | Java, C# |
| references/kotlin.md | Kotlin |
| references/php.md | PHP |
| references/javascript.md | JavaScript, TypeScript |
| references/python.md | Python |
| references/ruby.md | Ruby |
Running the Analyzer
The analyzer takes one file and detects the language from its extension. Always pass --warnings:
uv run {baseDir}/ct_analyzer/analyzer.py --warnings <source_file>
Without it the analyzer reports only error-severity findings, which means division, modulo and weak RNG. Four detector families are warning severity and stay silent: secret-dependent branches, early-exit comparison (memcmp, strcmp, .equals, ==), table lookups indexed by a secret, and variable-time encoding. Early-exit comparison of an authentication tag is the most common timing bug in real code — Lucky Thirteen was exactly that — so a default run is quiet about the finding you are most likely to have.
| Flag | Effect |
|---|---|
--warnings | Add the four warning-severity families above. Pass it every time |
--func <regex> | Restrict output to function names matching the regex |
--json | Machine-readable output |
--github | GitHub Actions annotations |
--arch <target> | Target architecture (x86_64, arm64, riscv64, ...) — native languages only |
--opt-level <level> | Optimization level (O0 through O3, Os, Oz) — native languages only |
--compiler <name> | Override compiler choice (gcc, clang, go, rustc, swiftc) |
Narrow a large file to the routines that handle secrets with a regex, for example --func 'sign|verify'.
Run natively compiled code (C, C++, Go, Rust, Swift) at more than one --arch and --opt-level. Division timing and branch lowering are architecture- and optimization-dependent: x86_64 IDIV and arm64 SDIV differ, and a cmov at -O2 can become a branch at -O0. A single clean run proves one configuration safe, not the code.
How --arch crosses depends on the toolchain. clang crosses with --target and needs no second compiler, but any source that includes libc headers also needs that target's C library headers — libc6-dev-riscv64-cross and friends — or it fails with bits/libc-header-start.h file not found. Go cross-builds through GOARCH, though go tool objdump has no riscv64 disassembler. A GNU cross toolchain is a separate binary, so gcc needs it named explicitly — --compiler x86_64-linux-gnu-gcc, --compiler riscv64-linux-gnu-gcc — and nothing is substituted for you, so the report always names the binary that ran. rustc needs the target's standard library (rustup target add), and Swift on Linux targets only the host. Compare against the toolchain that builds your product, not whichever cross build a distribution packages.
Re-run the whole sweep on the fix, across compilers, targets and every level including Os and Oz. Any fix that works by handing the compiler a constant divisor to strength-reduce is a fix only where the compiler chooses to cooperate, and that choice varies more than it looks. Replacing key_coef / (2 * gamma2) with a #defined divisor still emits a real divide here:
| Toolchain | Levels that emit a division |
|---|---|
| gcc riscv64 | O0 through Oz — every level |
| gcc arm64, gcc x86_64 | Os, Oz |
| clang arm64 | O0, Oz |
Strength reduction is an optimizer courtesy, not a language guarantee. Prefer an explicit multiply-shift, and verify it against the original expression over the full input range rather than on sampled values — an off-by-a-power-of-two reciprocal matches for millions of inputs before it diverges.
Java, Kotlin, and C# compile to JVM/CIL bytecode. The analyzer reads that bytecode, so --arch and --opt-level do not apply and the JIT may still introduce variable-time native code the analyzer cannot see.
Per-language coverage limits
Coverage is not uniform, and the gaps change what a clean report means:
| Language | What the report does not cover |
|---|---|
| Go | Only symbols from the analyzed file. go build links the runtime in, and its divisions — all on public data — would otherwise dominate the findings |
| JavaScript, TypeScript | Bytecode findings are restricted to functions the file declares by name, because V8 dumps node's internals the same way it dumps yours. Anonymous callbacks fall to the source scan. For TypeScript, bytecode findings name the function but carry no line, since V8's positions index the transpiled output |
| Python, Ruby, PHP | Bytecode reflects the interpreter that ran, not a JIT'd or alternative runtime |
| Rust | Analyzed as a library unless the file declares fn main; private functions with no caller may be optimized away before analysis |
| Swift | Targets the host platform on Linux; iOS and macOS triples need an Apple toolchain |
Since findings and silence both depend on the configuration, say which compiler, architecture, and optimization level produced a result when reporting it.
To sweep a directory, loop in the shell — the analyzer is a deterministic script, one invocation per file:
for f in src/crypto/*.c; do uv run {baseDir}/ct_analyzer/analyzer.py --warnings --json "$f"; done
Prerequisites
| Language | Requirement |
|---|---|
| C, C++, Go, Rust | gcc/clang, go, rustc in PATH |
| Swift | Xcode or Swift toolchain (swiftc) |
| Java / Kotlin | JDK (javac, javap); Kotlin also needs kotlinc |
| C# | .NET SDK plus ilspycmd (dotnet tool install -g ilspycmd) |
| PHP | PHP with the VLD extension or OPcache |
| JavaScript / TypeScript | Node.js |
| Python | Python 3.x |
| Ruby | Ruby with --dump=insns support |
On a "toolchain not found" error, see references/vm-compiled.md for JVM and .NET installation, macOS keg-only PATH configuration, and troubleshooting.
Interpreting Results
PASSED — no error-severity finding for the configuration you ran. Warnings do not affect it, so Result: PASSED alongside Warnings: 6 is normal and is not a clean result. Read the warning list before concluding anything.
FAILED — dangerous instructions found, reported per function:
[ERROR] SDIV
Function: decompose_vulnerable
Reason: SDIV has early termination optimization; execution time depends on operand values
Triaging Findings
The analyzer has no data flow analysis. It flags every dangerous instruction regardless of whether a secret reaches it, so a FAILED report is a worklist, not a verdict. Reporting the raw output as a set of vulnerabilities is the primary failure mode of this skill.
For each flagged instruction, read the source and answer one question: does an operand depend on secret data? Trace from the instruction's function back to the caller's inputs, then classify:
// FALSE POSITIVE: operands are a buffer length, already public from the ciphertext size
int num_blocks = data_len / 16;
// TRUE POSITIVE: dividend is a private-key coefficient; IDIV/SDIV leaks its magnitude
int32_t q = secret_coef / GAMMA2;
| Question | If yes |
|---|---|
| Is the operand a compile-time constant? | Likely false positive |
| Is the operand a public parameter — length, count, index bound? | Likely false positive |
| Is the operand derived from a key, plaintext, nonce, or token? | True positive |
| Can an attacker influence the operand's value? | True positive |
State the verdict and the data flow that justifies it for every flagged item. A finding you cannot trace to a secret is not a finding; say so explicitly rather than dropping it silently.
{baseDir}/ct_analyzer/tests/triage_samples/ holds a known-answer case per language: each fixture pairs a true positive with a false positive that the analyzer reports identically, and expectations.json records which is which and why. triage_c.c is the shortest example — the analyzer flags the division in both ct_high_bits and ct_block_count, and correct triage confirms the first and clears the second.
Weak-RNG and encoding findings ask a different question. For Math.random, mt_rand, random.randint, System.Random and base64_encode, no operand is secret, so "does an operand depend on a secret?" does not resolve them. Ask instead what the result is used for: seeding a nonce or key is a true positive, jittering a retry delay is not. These are reported by a regex scan over the source rather than from bytecode, so they are attributed to <source> with a line number instead of to the enclosing function — except in PHP, where they carry the function.
Comparison and lookup findings have their own question, and their own fix. For an early-exit comparison, ask whether either side is secret: comparing an authentication tag, MAC, or password hash is a true positive, comparing a public protocol header is not. For a table lookup, ask whether the index is secret — the array's contents do not matter, only what selects the element. Both are exploitable as written, so a confirmed one needs the language's constant-time primitive rather than a rewrite of the loop:
| Language | Constant-time comparison |
|---|---|
| C, C++ | CRYPTO_memcmp (OpenSSL) or sodium_memcmp |
| Go | crypto/subtle.ConstantTimeCompare |
| Rust | the subtle crate's ConstantTimeEq |
| Java, Kotlin | MessageDigest.isEqual |
| C# | CryptographicOperations.FixedTimeEquals |
| PHP | hash_equals |
| Python | hmac.compare_digest |
| Ruby | OpenSSL.secure_compare |
| JavaScript, TypeScript | crypto.timingSafeEqual |
A secret-indexed lookup has no drop-in replacement: it needs a bit-sliced or arithmetic formulation that touches every element, which is why AES S-box tables are the classic case. Encoding a secret through a table — base64_encode, bin2hex, chr/ord — is the same problem in a library, and paragonie/constant_time_encoding is the reference fix for PHP.
Limitations
- Static only — reads assembly and bytecode, never runtime behavior. Cache timing and other microarchitectural channels are invisible.
- No data flow analysis — see triage above.
- Configuration-specific — a different compiler, optimization level, architecture, or runtime version can emit different instructions from identical source.
Real-World Impact
- KyberSlash (2023) — division instructions in ML-KEM implementations allowed key recovery
- Lucky Thirteen (2013) — timing differences in CBC padding validation enabled plaintext recovery
- RSA timing attacks — early implementations leaked private key bits through division timing
References
- Cryptocoding Guidelines — defensive coding for crypto
- KyberSlash — division timing in post-quantum crypto
- BearSSL Constant-Time — practical constant-time techniques
Constant-Time Testing
Timing attacks exploit variations in execution time to extract secret information from cryptographic implementations. Unlike cryptanalysis that targets theoretical weaknesses, timing attacks leverage implementation flaws - and they can affect any cryptographic code.
Background
Timing attacks were introduced by Kocher in 1996. Since then, researchers have demonstrated practical attacks on RSA (Schindler), OpenSSL (Brumley and Boneh), AES implementations, and even post-quantum algorithms like Kyber.
Key Concepts
| Concept | Description |
|---|---|
| Constant-time | Code path and memory accesses independent of secret data |
| Timing leakage | Observable execution time differences correlated with secrets |
| Side channel | Information extracted from implementation rather than algorithm |
| Microarchitecture | CPU-level timing differences (cache, division, shifts) |
Why This Matters
Timing vulnerabilities can:
- Expose private keys - Extract secret exponents in RSA/ECDH
- Enable remote attacks - Network-observable timing differences
- Bypass cryptographic security - Undermine theoretical guarantees
- Persist silently - Often undetected without specialized analysis
Two prerequisites enable exploitation:
- Access to oracle - Sufficient queries to the vulnerable implementation
- Timing dependency - Correlation between execution time and secret data
Common Constant-Time Violation Patterns
Four patterns account for most timing vulnerabilities:
// 1. Conditional jumps - most severe timing differences
if(secret == 1) { ... }
while(secret > 0) { ... }
// 2. Array access - cache-timing attacks
lookup_table[secret];
// 3. Integer division (processor dependent)
data = secret / m;
// 4. Shift operation (processor dependent)
data = a << secret;
Conditional jumps cause different code paths, leading to vast timing differences.
Array access dependent on secrets enables cache-timing attacks, as shown in AES cache-timing research.
Integer division and shift operations leak secrets on certain CPU architectures and compiler configurations.
When patterns cannot be avoided, employ masking techniques to remove correlation between timing and secrets.
Example: Modular Exponentiation Timing Attacks
Modular exponentiation (used in RSA and Diffie-Hellman) is susceptible to timing attacks. RSA decryption computes:
$$ct^{d} \mod{N}$$
where $d$ is the secret exponent. The exponentiation by squaring optimization reduces multiplications to $\log{d}$:
$$ \begin{align*} & \textbf{Input: } \text{base }y,\text{exponent } d={d_n,\cdots,d_0}_2,\text{modulus } N \ & r = 1 \ & \textbf{for } i=|n| \text{ downto } 0: \ & \quad\textbf{if } d_i == 1: \ & \quad\quad r = r * y \mod{N} \ & \quad y = y * y \mod{N} \ & \textbf{return }r \end{align*} $$
The code branches on exponent bit $d_i$, violating constant-time principles. When $d_i = 1$, an additional multiplication occurs, increasing execution time and leaking bit information.
Montgomery multiplication (commonly used for modular arithmetic) also leaks timing: when intermediate values exceed modulus $N$, an additional reduction step is required. An attacker constructs inputs $y$ and $y'$ such that:
$$ \begin{align*} y^2 < y^3 < N \ y'^2 < N \leq y'^3 \end{align*} $$
For $y$, both multiplications take time $t_1+t_1$. For $y'$, the second multiplication requires reduction, taking time $t_1+t_2$. This timing difference reveals whether $d_i$ is 0 or 1.
When to Use
Apply constant-time analysis when:
- Auditing cryptographic implementations (primitives, protocols)
- Code handles secret keys, passwords, or sensitive cryptographic material
- Implementing crypto algorithms from scratch
- Reviewing PRs that touch crypto code
- Investigating potential timing vulnerabilities
Consider alternatives when:
- Code does not process secret data
- Public algorithms with no secret inputs
- Non-cryptographic timing requirements (performance optimization)
Quick Reference
| Scenario | Recommended Approach | Skill |
|---|---|---|
| Prove absence of leaks | Formal verification | SideTrail, ct-verif, FaCT |
| Detect statistical timing differences | Statistical testing | dudect |
| Track secret data flow at runtime | Dynamic analysis | timecop |
| Find cache-timing vulnerabilities | Symbolic execution | Binsec, pitchfork |
Constant-Time Tooling Categories
The cryptographic community has developed four categories of timing analysis tools:
| Category | Approach | Pros | Cons |
|---|---|---|---|
| Formal | Mathematical proof on model | Guarantees absence of leaks | Complexity, modeling assumptions |
| Symbolic | Symbolic execution paths | Concrete counterexamples | Time-intensive path exploration |
| Dynamic | Runtime tracing with marked secrets | Granular, flexible | Limited coverage to executed paths |
| Statistical | Measure real execution timing | Practical, simple setup | No root cause, noise sensitivity |
1. Formal Tools
Formal verification mathematically proves timing properties on an abstraction (model) of code. Tools create a model from source/binary and verify it satisfies specified properties (e.g., variables annotated as secret).
Popular tools:
Strengths: Proof of absence, language-agnostic (LLVM bytecode) Weaknesses: Requires expertise, modeling assumptions may miss real-world issues
2. Symbolic Tools
Symbolic execution analyzes how paths and memory accesses depend on symbolic variables (secrets). Provides concrete counterexamples. Focus on cache-timing attacks.
Popular tools:
Strengths: Concrete counterexamples aid debugging Weaknesses: Path explosion leads to long execution times
3. Dynamic Tools
Dynamic analysis marks sensitive memory regions and traces execution to detect timing-dependent operations.
Popular tools:
Strengths: Granular control, targeted analysis Weaknesses: Coverage limited to executed paths
Detailed Guidance: See the timecop skill for setup and usage.
4. Statistical Tools
Execute code with various inputs, measure elapsed time, and detect inconsistencies. Tests actual implementation including compiler optimizations and architecture.
Popular tools:
- dudect (see below)
- tlsfuzzer
Strengths: Simple setup, practical real-world results Weaknesses: No root cause info, noise obscures weak signals
Detailed Guidance: See the dudect skill for setup and usage.
Testing Workflow
Phase 1: Static Analysis Phase 2: Statistical Testing
┌─────────────────┐ ┌─────────────────┐
│ Identify secret │ → │ Detect timing │
│ data flow │ │ differences │
│ Tool: ct-verif │ │ Tool: dudect │
└─────────────────┘ └─────────────────┘
↓ ↓
Phase 4: Root Cause Phase 3: Dynamic Tracing
┌─────────────────┐ ┌─────────────────┐
│ Pinpoint leak │ ← │ Track secret │
│ location │ │ propagation │
│ Tool: Timecop │ │ Tool: Timecop │
└─────────────────┘ └─────────────────┘
Recommended approach:
- Start with dudect - Quick statistical check for timing differences
- If leaks found - Use Timecop to pinpoint root cause
- For high-assurance - Apply formal verification (ct-verif, SideTrail)
- Continuous monitoring - Integrate dudect into CI pipeline
Tools and Approaches
Dudect - Statistical Analysis
Dudect measures execution time for two input classes (fixed vs random) and uses Welch's t-test to detect statistically significant differences.
Detailed Guidance: See the dudect skill for complete setup, usage patterns, and CI integration.
Quick Start for Constant-Time Analysis
#define DUDECT_IMPLEMENTATION
#include "dudect.h"
uint8_t do_one_computation(uint8_t *data) {
// Code to measure goes here
}
void prepare_inputs(dudect_config_t *c, uint8_t *input_data, uint8_t *classes) {
for (size_t i = 0; i < c->number_measurements; i++) {
classes[i] = randombit();
uint8_t *input = input_data + (size_t)i * c->chunk_size;
if (classes[i] == 0) {
// Fixed input class
} else {
// Random input class
}
}
}
Key advantages:
- Simple C header-only integration
- Statistical rigor via Welch's t-test
- Works with compiled binaries (real-world conditions)
Key limitations:
- No root cause information when leak detected
- Sensitive to measurement noise
- Cannot guarantee absence of leaks (statistical confidence only)
Timecop - Dynamic Tracing
Timecop wraps Valgrind to detect runtime operations dependent on secret memory regions.
Detailed Guidance: See the timecop skill for installation, examples, and debugging.
Quick Start for Constant-Time Analysis
#include "valgrind/memcheck.h"
#define poison(addr, len) VALGRIND_MAKE_MEM_UNDEFINED(addr, len)
#define unpoison(addr, len) VALGRIND_MAKE_MEM_DEFINED(addr, len)
int main() {
unsigned long long secret_key = 0x12345678;
// Mark secret as poisoned
poison(&secret_key, sizeof(secret_key));
// Any branching or memory access dependent on secret_key
// will be reported by Valgrind
crypto_operation(secret_key);
unpoison(&secret_key, sizeof(secret_key));
}
Run with Valgrind:
valgrind --leak-check=full --track-origins=yes ./binary
Key advantages:
- Pinpoints exact line of timing leak
- No code instrumentation required
- Tracks secret propagation through execution
Key limitations:
- Cannot detect microarchitecture timing differences
- Coverage limited to executed paths
- Performance overhead (runs on synthetic CPU)
Implementation Guide
Phase 1: Initial Assessment
Identify cryptographic code handling secrets:
- Private keys, exponents, nonces
- Password hashes, authentication tokens
- Encryption/decryption operations
Quick statistical check:
- Write dudect harness for the crypto function
- Run for 5-10 minutes with
timeout 600 ./ct_test - Monitor t-value: high absolute values indicate leakage
Tools: dudect Expected time: 1-2 hours (harness writing + initial run)
Phase 2: Detailed Analysis
If dudect detects leakage:
Root cause investigation:
- Mark secret variables with Timecop
poison() - Run under Valgrind to identify exact line
- Review the four common violation patterns
- Check assembly output for conditional branches
Tools: Timecop, compiler output (objdump -d)
Phase 3: Remediation
Fix the timing leak:
- Replace conditional branches with constant-time selection (bitwise operations)
- Use constant-time comparison functions
- Replace array lookups with constant-time alternatives or masking
- Verify compiler doesn't optimize away constant-time code
Re-verify:
- Run dudect again for extended period (30+ minutes)
- Test across different compilers and optimization levels
- Test on different CPU architectures
Phase 4: Continuous Monitoring
Integrate into CI:
- Add dudect tests to test suite
- Run for fixed duration (5-10 minutes in CI)
- Fail build if leakage detected
See the dudect skill for CI integration examples.
Common Vulnerabilities
| Vulnerability | Description | Detection | Severity |
|---|---|---|---|
| Secret-dependent branch | if (secret_bit) { ... } | dudect, Timecop | CRITICAL |
| Secret-dependent array access | table[secret_index] | Timecop, Binsec | HIGH |
| Variable-time division | result = x / secret | Timecop | MEDIUM |
| Variable-time shift | result = x << secret | Timecop | MEDIUM |
| Montgomery reduction leak | Extra reduction when intermediate > N | dudect | HIGH |
Secret-Dependent Branch: Deep Dive
The vulnerability: Execution time differs based on whether branch is taken. Common in optimized modular exponentiation (square-and-multiply).
How to detect with dudect:
uint8_t do_one_computation(uint8_t *data) {
uint64_t base = ((uint64_t*)data)[0];
uint64_t exponent = ((uint64_t*)data)[1]; // Secret!
return mod_exp(base, exponent, MODULUS);
}
void prepare_inputs(dudect_config_t *c, uint8_t *input_data, uint8_t *classes) {
for (size_t i = 0; i < c->number_measurements; i++) {
classes[i] = randombit();
uint64_t *input = (uint64_t*)(input_data + i * c->chunk_size);
input[0] = rand(); // Random base
input[1] = (classes[i] == 0) ? FIXED_EXPONENT : rand(); // Fixed vs random
}
}
How to detect with Timecop:
poison(&exponent, sizeof(exponent));
result = mod_exp(base, exponent, modulus);
unpoison(&exponent, sizeof(exponent));
Valgrind will report:
Conditional jump or move depends on uninitialised value(s)
at 0x40115D: mod_exp (example.c:14)
Related skill: dudect, timecop
Case Studies
Case Study: OpenSSL RSA Timing Attack
Brumley and Boneh (2005) extracted RSA private keys from OpenSSL over a network. The vulnerability exploited Montgomery multiplication's variable-time reduction step.
Attack vector: Timing differences in modular exponentiation Detection approach: Statistical analysis (precursor to dudect) Impact: Remote key extraction
Tools used: Custom timing measurement Techniques applied: Statistical analysis, chosen-ciphertext queries
Case Study: KyberSlash
Post-quantum algorithm Kyber's reference implementation contained timing vulnerabilities in polynomial operations. Division operations leaked secret coefficients.
Attack vector: Secret-dependent division timing Detection approach: Dynamic analysis and statistical testing Impact: Secret key recovery in post-quantum cryptography
Tools used: Timing measurement tools Techniques applied: Differential timing analysis
Advanced Usage
Tips and Tricks
| Tip | Why It Helps |
|---|---|
Pin dudect to isolated CPU core (taskset -c 2) | Reduces OS noise, improves signal detection |
| Test multiple compilers (gcc, clang, MSVC) | Optimizations may introduce or remove leaks |
| Run dudect for extended periods (hours) | Increases statistical confidence |
| Minimize non-crypto code in harness | Reduces noise that masks weak signals |
Check assembly output (objdump -d) | Verify compiler didn't introduce branches |
Use -O3 -march=native in testing | Matches production optimization levels |
Common Mistakes
| Mistake | Why It's Wrong | Correct Approach |
|---|---|---|
| Only testing one input distribution | May miss leaks visible with other patterns | Test fixed-vs-random, fixed-vs-fixed-different, etc. |
| Short dudect runs (< 1 minute) | Insufficient measurements for weak signals | Run 5-10+ minutes, longer for high assurance |
| Ignoring compiler optimization levels | -O0 may hide leaks present in -O3 | Test at production optimization level |
| Not testing on target architecture | x86 vs ARM have different timing characteristics | Test on deployment platform |
| Marking too much as secret in Timecop | False positives, unclear results | Mark only true secrets (keys, not public data) |
Related Skills
Tool Skills
| Skill | Primary Use in Constant-Time Analysis |
|---|---|
| dudect | Statistical detection of timing differences via Welch's t-test |
| timecop | Dynamic tracing to pinpoint exact location of timing leaks |
Technique Skills
| Skill | When to Apply |
|---|---|
| coverage-analysis | Ensure test inputs exercise all code paths in crypto function |
| ci-integration | Automate constant-time testing in continuous integration pipeline |
Related Domain Skills
| Skill | Relationship |
|---|---|
| crypto-testing | Constant-time analysis is essential component of cryptographic testing |
| fuzzing | Fuzzing crypto code may trigger timing-dependent paths |
Skill Dependency Map
┌─────────────────────────┐
│ constant-time-analysis │
│ (this skill) │
└───────────┬─────────────┘
│
┌───────────────┴───────────────┐
│ │
▼ ▼
┌───────────────────┐ ┌───────────────────┐
│ dudect │ │ timecop │
│ (statistical) │ │ (dynamic) │
└────────┬──────────┘ └────────┬──────────┘
│ │
└───────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ Supporting Techniques │
│ coverage, CI integration │
└──────────────────────────────┘
Resources
Key External Resources
These results must be false: A usability evaluation of constant-time analysis tools Comprehensive usability study of constant-time analysis tools. Key findings: developers struggle with false positives, need better error messages, and benefit from tool integration. Evaluates FaCT, ct-verif, dudect, and Memsan across multiple cryptographic implementations. Recommends improved tooling UX and better documentation.
List of constant-time tools - CROCS Curated catalog of constant-time analysis tools with tutorials. Covers formal tools (ct-verif, FaCT), dynamic tools (Memsan, Timecop), symbolic tools (Binsec), and statistical tools (dudect). Includes practical tutorials for setup and usage.
Paul Kocher: Timing Attacks on Implementations of Diffie-Hellman, RSA, DSS, and Other Systems Original 1996 paper introducing timing attacks. Demonstrates attacks on modular exponentiation in RSA and Diffie-Hellman. Essential historical context for understanding timing vulnerabilities.
Remote Timing Attacks are Practical (Brumley & Boneh) Demonstrates practical remote timing attacks against OpenSSL. Shows network-level timing differences are sufficient to extract RSA keys. Proves timing attacks work in realistic network conditions.
Cache-timing attacks on AES Shows AES implementations using lookup tables are vulnerable to cache-timing attacks. Demonstrates practical attacks extracting AES keys via cache timing side channels.
KyberSlash: Division Timings Leak Secrets Recent discovery of timing vulnerabilities in Kyber (NIST post-quantum standard). Shows division operations leak secret coefficients. Highlights that constant-time issues persist even in modern post-quantum cryptography.
Video Resources
- Trail of Bits: Constant-Time Programming - Overview of constant-time programming principles and tools
Coverage Analysis
Coverage analysis is essential for understanding which parts of your code are exercised during fuzzing. It helps identify fuzzing blockers like magic value checks and tracks the effectiveness of harness improvements over time.
Overview
Code coverage during fuzzing serves two critical purposes:
- Assessing harness effectiveness: Understand which parts of your application are actually executed by your fuzzing harnesses
- Tracking fuzzing progress: Monitor how coverage changes when updating harnesses, fuzzers, or the system under test (SUT)
Coverage is a proxy for fuzzer capability and performance. While coverage is not ideal for measuring fuzzer performance in absolute terms, it reliably indicates whether your harness works effectively in a given setup.
Key Concepts
| Concept | Description |
|---|---|
| Coverage instrumentation | Compiler flags that track which code paths are executed |
| Corpus coverage | Coverage achieved by running all test cases in a fuzzing corpus |
| Magic value checks | Hard-to-discover conditional checks that block fuzzer progress |
| Coverage-guided fuzzing | Fuzzing strategy that prioritizes inputs that discover new code paths |
| Coverage report | Visual or textual representation of executed vs. unexecuted code |
When to Apply
Apply this technique when:
- Starting a new fuzzing campaign to establish a baseline
- Fuzzer appears to plateau without finding new paths
- After harness modifications to verify improvements
- When migrating between different fuzzers
- Identifying areas requiring dictionary entries or seed inputs
- Debugging why certain code paths aren't reached
Skip this technique when:
- Fuzzing campaign is actively finding crashes
- Coverage infrastructure isn't set up yet
- Working with extremely large codebases where full coverage reports are impractical
- Fuzzer's internal coverage metrics are sufficient for your needs
Quick Reference
| Task | Command/Pattern |
|---|---|
| LLVM coverage instrumentation (C/C++) | -fprofile-instr-generate -fcoverage-mapping |
| GCC coverage instrumentation | -ftest-coverage -fprofile-arcs |
| cargo-fuzz coverage (Rust) | cargo +nightly fuzz coverage <target> |
| Generate LLVM profile data | llvm-profdata merge -sparse file.profraw -o file.profdata |
| LLVM coverage report | llvm-cov report ./binary -instr-profile=file.profdata |
| LLVM HTML report | llvm-cov show ./binary -instr-profile=file.profdata -format=html -output-dir html/ |
| gcovr HTML report | gcovr --html-details -o coverage.html |
Ideal Coverage Workflow
The following workflow represents best practices for integrating coverage analysis into your fuzzing campaigns:
[Fuzzing Campaign]
|
v
[Generate Corpus]
|
v
[Coverage Analysis]
|
+---> Coverage Increased? --> Continue fuzzing with larger corpus
|
+---> Coverage Decreased? --> Fix harness or investigate SUT changes
|
+---> Coverage Plateaued? --> Add dictionary entries or seed inputs
Key principle: Use the corpus generated after each fuzzing campaign to calculate coverage, rather than real-time fuzzer statistics. This approach provides reproducible, comparable measurements across different fuzzing tools.
Step-by-Step
Step 1: Build with Coverage Instrumentation
Choose your instrumentation method based on toolchain:
LLVM/Clang (C/C++):
clang++ -fprofile-instr-generate -fcoverage-mapping \
-O2 -DNO_MAIN \
main.cc harness.cc execute-rt.cc -o fuzz_exec
GCC (C/C++):
g++ -ftest-coverage -fprofile-arcs \
-O2 -DNO_MAIN \
main.cc harness.cc execute-rt.cc -o fuzz_exec_gcov
Rust:
rustup toolchain install nightly --component llvm-tools-preview
cargo +nightly fuzz coverage fuzz_target_1
Step 2: Create Execution Runtime (C/C++ only)
For C/C++ projects, create a runtime that executes your corpus:
// execute-rt.cc
#include <stdio.h>
#include <stdlib.h>
#include <dirent.h>
#include <stdint.h>
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size);
void load_file_and_test(const char *filename) {
FILE *file = fopen(filename, "rb");
if (file == NULL) {
printf("Failed to open file: %s\n", filename);
return;
}
fseek(file, 0, SEEK_END);
long filesize = ftell(file);
rewind(file);
uint8_t *buffer = (uint8_t*) malloc(filesize);
if (buffer == NULL) {
printf("Failed to allocate memory for file: %s\n", filename);
fclose(file);
return;
}
long read_size = (long) fread(buffer, 1, filesize, file);
if (read_size != filesize) {
printf("Failed to read file: %s\n", filename);
free(buffer);
fclose(file);
return;
}
LLVMFuzzerTestOneInput(buffer, filesize);
free(buffer);
fclose(file);
}
int main(int argc, char **argv) {
if (argc != 2) {
printf("Usage: %s <directory>\n", argv[0]);
return 1;
}
DIR *dir = opendir(argv[1]);
if (dir == NULL) {
printf("Failed to open directory: %s\n", argv[1]);
return 1;
}
struct dirent *entry;
while ((entry = readdir(dir)) != NULL) {
if (entry->d_type == DT_REG) {
char filepath[1024];
snprintf(filepath, sizeof(filepath), "%s/%s", argv[1], entry->d_name);
load_file_and_test(filepath);
}
}
closedir(dir);
return 0;
}
Step 3: Execute on Corpus
LLVM (C/C++):
LLVM_PROFILE_FILE=fuzz.profraw ./fuzz_exec corpus/
GCC (C/C++):
./fuzz_exec_gcov corpus/
Rust:
Coverage data is automatically generated when running cargo fuzz coverage.
Step 4: Process Coverage Data
LLVM:
# Merge raw profile data
llvm-profdata merge -sparse fuzz.profraw -o fuzz.profdata
# Generate text report
llvm-cov report ./fuzz_exec \
-instr-profile=fuzz.profdata \
-ignore-filename-regex='harness.cc|execute-rt.cc'
# Generate HTML report
llvm-cov show ./fuzz_exec \
-instr-profile=fuzz.profdata \
-ignore-filename-regex='harness.cc|execute-rt.cc' \
-format=html -output-dir fuzz_html/
GCC with gcovr:
# Install gcovr (uv manages the isolated environment)
uv tool install gcovr
export PATH="$HOME/.local/bin:$PATH" # uv's tool bin, if not already present
# Generate report
gcovr --gcov-executable "llvm-cov gcov" \
--exclude harness.cc --exclude execute-rt.cc \
--root . --html-details -o coverage.html
Rust:
# Install required tools
cargo install cargo-binutils rustfilt
# Create HTML generation script
cat <<'EOF' > ./generate_html
#!/bin/sh
if [ $# -lt 1 ]; then
echo "Error: Name of fuzz target is required."
echo "Usage: $0 fuzz_target [sources...]"
exit 1
fi
FUZZ_TARGET="$1"
shift
SRC_FILTER="$@"
TARGET=$(rustc -vV | sed -n 's|host: ||p')
cargo +nightly cov -- show -Xdemangler=rustfilt \
"target/$TARGET/coverage/$TARGET/release/$FUZZ_TARGET" \
-instr-profile="fuzz/coverage/$FUZZ_TARGET/coverage.profdata" \
-show-line-counts-or-regions -show-instantiations \
-format=html -o fuzz_html/ $SRC_FILTER
EOF
chmod +x ./generate_html
# Generate HTML report
./generate_html fuzz_target_1 src/lib.rs
Step 5: Analyze Results
Review the coverage report to identify:
- Uncovered code blocks: Areas that may need better seed inputs or dictionary entries
- Magic value checks: Conditional statements with hardcoded values that block progress
- Dead code: Functions that may not be reachable through your harness
- Coverage changes: Compare against baseline to track improvements or regressions
Common Patterns
Pattern: Identifying Magic Values
Problem: Fuzzer cannot discover paths guarded by magic value checks.
Coverage reveals:
// Coverage shows this block is never executed
if (buf == 0x7F454C46) { // ELF magic number
// start parsing buf
}
Solution: Add magic values to dictionary file:
# magic.dict
"\x7F\x45\x4C\x46"
Pattern: Handling Crashing Inputs
Problem: Coverage generation fails when corpus contains crashing inputs.
Before:
./fuzz_exec corpus/ # Crashes on bad input, no coverage generated
After:
// Fork before executing to isolate crashes
int main(int argc, char **argv) {
// ... directory opening code ...
while ((entry = readdir(dir)) != NULL) {
if (entry->d_type == DT_REG) {
pid_t pid = fork();
if (pid == 0) {
// Child process - crash won't affect parent
char filepath[1024];
snprintf(filepath, sizeof(filepath), "%s/%s", argv[1], entry->d_name);
load_file_and_test(filepath);
exit(0);
} else {
// Parent waits for child
waitpid(pid, NULL, 0);
}
}
}
}
Pattern: CMake Integration
Use Case: Adding coverage builds to CMake projects.
project(FuzzingProject)
cmake_minimum_required(VERSION 3.0)
# Main binary
add_executable(program main.cc)
# Fuzzing binary
add_executable(fuzz main.cc harness.cc)
target_compile_definitions(fuzz PRIVATE NO_MAIN=1)
target_compile_options(fuzz PRIVATE -g -O2 -fsanitize=fuzzer)
target_link_libraries(fuzz -fsanitize=fuzzer)
# Coverage execution binary
add_executable(fuzz_exec main.cc harness.cc execute-rt.cc)
target_compile_definitions(fuzz_exec PRIVATE NO_MAIN)
target_compile_options(fuzz_exec PRIVATE -O2 -fprofile-instr-generate -fcoverage-mapping)
target_link_libraries(fuzz_exec -fprofile-instr-generate)
Build:
cmake -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ .
cmake --build . --target fuzz_exec
Advanced Usage
Tips and Tricks
| Tip | Why It Helps |
|---|---|
Use LLVM 18+ with -show-directory-coverage | Organizes large reports by directory structure instead of flat file list |
| Export to lcov format for better HTML | llvm-cov export -format=lcov + genhtml provides cleaner per-file reports |
| Compare coverage across campaigns | Store .profdata files with timestamps to track progress over time |
| Filter harness code from reports | Use -ignore-filename-regex to focus on SUT coverage only |
| Automate coverage in CI/CD | Generate coverage reports automatically after scheduled fuzzing runs |
| Use gcovr 5.1+ for Clang 14+ | Older gcovr versions have compatibility issues with recent LLVM |
Incremental Coverage Updates
GCC's gcov instrumentation incrementally updates .gcda files across multiple runs. This is useful for tracking coverage as you add test cases:
# First run
./fuzz_exec_gcov corpus_batch_1/
gcovr --html coverage_v1.html
# Second run (adds to existing coverage)
./fuzz_exec_gcov corpus_batch_2/
gcovr --html coverage_v2.html
# Start fresh
gcovr --delete # Remove .gcda files
./fuzz_exec_gcov corpus/
Handling Large Codebases
For projects with hundreds of source files:
-
Filter by prefix: Only generate reports for relevant directories
llvm-cov show ./fuzz_exec -instr-profile=fuzz.profdata /path/to/src/ -
Use directory coverage: Group by directory to reduce clutter (LLVM 18+)
llvm-cov show -show-directory-coverage -format=html -output-dir html/ -
Generate JSON for programmatic analysis:
llvm-cov export -format=lcov > coverage.json
Differential Coverage
Compare coverage between two fuzzing campaigns:
# Campaign 1
LLVM_PROFILE_FILE=campaign1.profraw ./fuzz_exec corpus1/
llvm-profdata merge -sparse campaign1.profraw -o campaign1.profdata
# Campaign 2
LLVM_PROFILE_FILE=campaign2.profraw ./fuzz_exec corpus2/
llvm-profdata merge -sparse campaign2.profraw -o campaign2.profdata
# Compare
llvm-cov show ./fuzz_exec \
-instr-profile=campaign2.profdata \
-instr-profile=campaign1.profdata \
-show-line-counts-or-regions
Anti-Patterns
| Anti-Pattern | Problem | Correct Approach |
|---|---|---|
| Using fuzzer-reported coverage for comparisons | Different fuzzers calculate coverage differently, making cross-tool comparison meaningless | Use dedicated coverage tools (llvm-cov, gcovr) for reproducible measurements |
| Generating coverage with optimizations | -O3 optimizations can eliminate code, making coverage misleading | Use -O2 or -O0 for coverage builds |
| Not filtering harness code | Harness coverage inflates numbers and obscures SUT coverage | Use -ignore-filename-regex or --exclude to filter harness files |
| Mixing LLVM and GCC instrumentation | Incompatible formats cause parsing failures | Stick to one toolchain for coverage builds |
| Ignoring crashing inputs | Crashes prevent coverage generation, hiding real coverage data | Fix crashes first, or use process forking to isolate them |
| Not tracking coverage over time | One-time coverage checks miss regressions and improvements | Store coverage data with timestamps and track trends |
Tool-Specific Guidance
libFuzzer
libFuzzer uses LLVM's SanitizerCoverage by default for guiding fuzzing, but you need separate instrumentation for generating reports.
Build for coverage:
clang++ -fprofile-instr-generate -fcoverage-mapping \
-O2 -DNO_MAIN \
main.cc harness.cc execute-rt.cc -o fuzz_exec
Execute corpus and generate report:
LLVM_PROFILE_FILE=fuzz.profraw ./fuzz_exec corpus/
llvm-profdata merge -sparse fuzz.profraw -o fuzz.profdata
llvm-cov show ./fuzz_exec -instr-profile=fuzz.profdata -format=html -output-dir html/
Integration tips:
- Don't use
-fsanitize=fuzzerfor coverage builds (it conflicts with profile instrumentation) - Reuse the same harness function (
LLVMFuzzerTestOneInput) with a different main function - Use the
-ignore-filename-regexflag to exclude harness code from coverage reports - Consider using llvm-cov's
-show-instantiationflag for template-heavy C++ code
AFL++
AFL++ provides its own coverage feedback mechanism, but for detailed reports use standard LLVM/GCC tools.
Build for coverage with LLVM:
clang++ -fprofile-instr-generate -fcoverage-mapping \
-O2 main.cc harness.cc execute-rt.cc -o fuzz_exec
Build for coverage with GCC:
AFL_USE_ASAN=0 afl-gcc -ftest-coverage -fprofile-arcs \
main.cc harness.cc execute-rt.cc -o fuzz_exec_gcov
Execute and generate report:
# LLVM approach
LLVM_PROFILE_FILE=fuzz.profraw ./fuzz_exec afl_output/queue/
llvm-profdata merge -sparse fuzz.profraw -o fuzz.profdata
llvm-cov report ./fuzz_exec -instr-profile=fuzz.profdata
# GCC approach
./fuzz_exec_gcov afl_output/queue/
gcovr --html-details -o coverage.html
Integration tips:
- Don't use AFL++'s instrumentation (
afl-clang-fast) for coverage builds - Use standard compilers with coverage flags instead
- AFL++'s
queue/directory contains your corpus - AFL++'s built-in coverage statistics are useful for real-time monitoring but not for detailed analysis
cargo-fuzz (Rust)
cargo-fuzz provides built-in coverage generation using LLVM tools.
Install prerequisites:
rustup toolchain install nightly --component llvm-tools-preview
cargo install cargo-binutils rustfilt
Generate coverage data:
cargo +nightly fuzz coverage fuzz_target_1
Create HTML report script:
cat <<'EOF' > ./generate_html
#!/bin/sh
FUZZ_TARGET="$1"
shift
SRC_FILTER="$@"
TARGET=$(rustc -vV | sed -n 's|host: ||p')
cargo +nightly cov -- show -Xdemangler=rustfilt \
"target/$TARGET/coverage/$TARGET/release/$FUZZ_TARGET" \
-instr-profile="fuzz/coverage/$FUZZ_TARGET/coverage.profdata" \
-show-line-counts-or-regions -show-instantiations \
-format=html -o fuzz_html/ $SRC_FILTER
EOF
chmod +x ./generate_html
Generate report:
./generate_html fuzz_target_1 src/lib.rs
Integration tips:
- Always use the nightly toolchain for coverage
- The
-Xdemangler=rustfiltflag makes function names readable - Filter by source files (e.g.,
src/lib.rs) to focus on crate code - Use
-show-line-counts-or-regionsand-show-instantiationsfor better Rust-specific output - Corpus is located in
fuzz/corpus/<target>/
honggfuzz
honggfuzz works with standard LLVM/GCC coverage instrumentation.
Build for coverage:
# Use standard compiler, not honggfuzz compiler
clang -fprofile-instr-generate -fcoverage-mapping \
-O2 harness.c execute-rt.c -o fuzz_exec
Execute corpus:
LLVM_PROFILE_FILE=fuzz.profraw ./fuzz_exec honggfuzz_workspace/
Integration tips:
- Don't use
hfuzz-clangfor coverage builds - honggfuzz corpus is typically in a workspace directory
- Use the same LLVM workflow as libFuzzer
Troubleshooting
| Issue | Cause | Solution |
|---|---|---|
error: no profile data available | Profile wasn't generated or wrong path | Verify LLVM_PROFILE_FILE was set and .profraw file exists |
Failed to load coverage | Mismatch between binary and profile data | Rebuild binary with same flags used during execution |
| Coverage reports show 0% | Wrong binary used for report generation | Use the instrumented binary, not the fuzzing binary |
no_working_dir_found error (gcovr) | .gcda files in unexpected location | Add --gcov-ignore-errors=no_working_dir_found flag |
| Crashes prevent coverage generation | Corpus contains crashing inputs | Filter crashes or use forking approach to isolate failures |
| Coverage decreases after harness change | Harness now skips certain code paths | Review harness logic; may need to support more input formats |
| HTML report is flat file list | Using older LLVM version | Upgrade to LLVM 18+ and use -show-directory-coverage |
incompatible instrumentation | Mixing LLVM and GCC coverage | Rebuild everything with same toolchain |
Related Skills
Tools That Use This Technique
| Skill | How It Applies |
|---|---|
| libfuzzer | Uses SanitizerCoverage for feedback; coverage analysis evaluates harness effectiveness |
| aflpp | Uses edge coverage for feedback; detailed analysis requires separate instrumentation |
| cargo-fuzz | Built-in cargo fuzz coverage command for Rust projects |
| honggfuzz | Uses edge coverage; analyze with standard LLVM/GCC tools |
Related Techniques
| Skill | Relationship |
|---|---|
| fuzz-harness-writing | Coverage reveals which code paths harness reaches; guides harness improvements |
| fuzzing-dictionaries | Coverage identifies magic value checks that need dictionary entries |
| corpus-management | Coverage analysis helps curate corpora by identifying redundant test cases |
| sanitizers | Coverage helps verify sanitizer-instrumented code is actually executed |
Resources
Key External Resources
LLVM Source-Based Code Coverage Comprehensive guide to LLVM's profile instrumentation, including advanced features like branch coverage, region coverage, and integration with existing build systems. Covers compiler flags, runtime behavior, and profile data formats.
llvm-cov Command Guide
Detailed CLI reference for llvm-cov commands including show, report, and export. Documents all filtering options, output formats, and integration with llvm-profdata.
gcovr Documentation Complete guide to gcovr tool for generating coverage reports from gcov data. Covers HTML themes, filtering options, multi-directory projects, and CI/CD integration patterns.
SanitizerCoverage Documentation Low-level documentation for LLVM's SanitizerCoverage instrumentation. Explains inline 8-bit counters, PC tables, and how fuzzers use coverage feedback for guidance.
On the Evaluation of Fuzzer Performance Research paper examining limitations of coverage as a fuzzing performance metric. Argues for more nuanced evaluation methods beyond simple code coverage percentages.
Video Resources
Not applicable - coverage analysis is primarily a tooling and workflow topic best learned through documentation and hands-on practice.
Fuzzing Dictionary
A fuzzing dictionary provides domain-specific tokens to guide the fuzzer toward interesting inputs. Instead of purely random mutations, the fuzzer incorporates known keywords, magic numbers, protocol commands, and format-specific strings that are more likely to reach deeper code paths in parsers, protocol handlers, and file format processors.
Overview
Dictionaries are text files containing quoted strings that represent meaningful tokens for your target. They help fuzzers bypass early validation checks and explore code paths that would be difficult to reach through blind mutation alone.
Key Concepts
| Concept | Description |
|---|---|
| Dictionary Entry | A quoted string (e.g., "keyword") or key-value pair (e.g., kw="value") |
| Hex Escapes | Byte sequences like "\xF7\xF8" for non-printable characters |
| Token Injection | Fuzzer inserts dictionary entries into generated inputs |
| Cross-Fuzzer Format | Dictionary files work with libFuzzer, AFL++, and cargo-fuzz |
When to Apply
Apply this technique when:
- Fuzzing parsers (JSON, XML, config files)
- Fuzzing protocol implementations (HTTP, DNS, custom protocols)
- Fuzzing file format handlers (PNG, PDF, media codecs)
- Coverage plateaus early without reaching deeper logic
- Target code checks for specific keywords or magic values
Skip this technique when:
- Fuzzing pure algorithms without format expectations
- Target has no keyword-based parsing
- Corpus already achieves high coverage
Quick Reference
| Task | Command/Pattern |
|---|---|
| Use with libFuzzer | ./fuzz -dict=./dictionary.dict ... |
| Use with AFL++ | afl-fuzz -x ./dictionary.dict ... |
| Use with cargo-fuzz | cargo fuzz run fuzz_target -- -dict=./dictionary.dict |
| Extract from header | grep -o '".*"' header.h > header.dict |
| Generate from binary | strings ./binary | sed 's/^/"&/; s/$/&"/' > strings.dict |
Step-by-Step
Step 1: Create Dictionary File
Create a text file with quoted strings on each line. Use comments (#) for documentation.
Example dictionary format:
# Lines starting with '#' and empty lines are ignored.
# Adds "blah" (w/o quotes) to the dictionary.
kw1="blah"
# Use \\ for backslash and \" for quotes.
kw2="\"ac\\dc\""
# Use \xAB for hex values
kw3="\xF7\xF8"
# the name of the keyword followed by '=' may be omitted:
"foo\x0Abar"
Step 2: Generate Dictionary Content
Choose a generation method based on what's available:
From LLM: Prompt ChatGPT or Claude with:
A dictionary can be used to guide the fuzzer. Write me a dictionary file for fuzzing a <PNG parser>. Each line should be a quoted string or key-value pair like kw="value". Include magic bytes, chunk types, and common header values. Use hex escapes like "\xF7\xF8" for binary values.
From header files:
grep -o '".*"' header.h > header.dict
From man pages (for CLI tools):
man curl | grep -oP '^\s*(--|-)\K\S+' | sed 's/[,.]$//' | sed 's/^/"&/; s/$/&"/' | sort -u > man.dict
From binary strings:
strings ./binary | sed 's/^/"&/; s/$/&"/' > strings.dict
Step 3: Pass Dictionary to Fuzzer
Use the appropriate flag for your fuzzer (see Quick Reference above).
Common Patterns
Pattern: Protocol Keywords
Use Case: Fuzzing HTTP or custom protocol handlers
Dictionary content:
# HTTP methods
"GET"
"POST"
"PUT"
"DELETE"
"HEAD"
# Headers
"Content-Type"
"Authorization"
"Host"
# Protocol markers
"HTTP/1.1"
"HTTP/2.0"
Pattern: Magic Bytes and File Format Headers
Use Case: Fuzzing image parsers, media decoders, archive handlers
Dictionary content:
# PNG magic bytes and chunks
png_magic="\x89PNG\r\n\x1a\n"
ihdr="IHDR"
plte="PLTE"
idat="IDAT"
iend="IEND"
# JPEG markers
jpeg_soi="\xFF\xD8"
jpeg_eoi="\xFF\xD9"
Pattern: Configuration File Keywords
Use Case: Fuzzing config file parsers (YAML, TOML, INI)
Dictionary content:
# Common config keywords
"true"
"false"
"null"
"version"
"enabled"
"disabled"
# Section headers
"[general]"
"[network]"
"[security]"
Advanced Usage
Tips and Tricks
| Tip | Why It Helps |
|---|---|
| Combine multiple generation methods | LLM-generated keywords + strings from binary covers broad surface |
| Include boundary values | "0", "-1", "2147483647" trigger edge cases |
| Add format delimiters | :, =, {, } help fuzzer construct valid structures |
| Keep dictionaries focused | 50-200 entries perform better than thousands |
| Test dictionary effectiveness | Run with and without dict, compare coverage |
Auto-Generated Dictionaries (AFL++)
When using afl-clang-lto compiler, AFL++ automatically extracts dictionary entries from string comparisons in the binary. This happens at compile time via the AUTODICTIONARY feature.
Enable auto-dictionary:
export AFL_LLVM_DICT2FILE=auto.dict
afl-clang-lto++ target.cc -o target
# Dictionary saved to auto.dict
afl-fuzz -x auto.dict -i in -o out -- ./target
Combining Multiple Dictionaries
Some fuzzers support multiple dictionary files:
# AFL++ with multiple dictionaries
afl-fuzz -x keywords.dict -x formats.dict -i in -o out -- ./target
Anti-Patterns
| Anti-Pattern | Problem | Correct Approach |
|---|---|---|
| Including full sentences | Fuzzer needs atomic tokens, not prose | Break into individual keywords |
| Duplicating entries | Wastes mutation budget | Use sort -u to deduplicate |
| Over-sized dictionaries | Slows fuzzer, dilutes useful tokens | Keep focused: 50-200 most relevant entries |
| Missing hex escapes | Non-printable bytes become mangled | Use \xXX for binary values |
| No comments | Hard to maintain and audit | Document sections with # comments |
Tool-Specific Guidance
libFuzzer
clang++ -fsanitize=fuzzer,address harness.cc -o fuzz
./fuzz -dict=./dictionary.dict corpus/
Integration tips:
- Dictionary tokens are inserted/replaced during mutations
- Combine with
-max_lento control input size - Use
-print_final_stats=1to see dictionary effectiveness metrics - Dictionary entries longer than
-max_lenare ignored
AFL++
afl-fuzz -x ./dictionary.dict -i input/ -o output/ -- ./target @@
Integration tips:
- AFL++ supports multiple
-xflags for multiple dictionaries - Use
AFL_LLVM_DICT2FILEwithafl-clang-ltofor auto-generated dictionaries - Dictionary effectiveness shown in fuzzer stats UI
- Tokens are used during deterministic and havoc stages
cargo-fuzz (Rust)
cargo fuzz run fuzz_target -- -dict=./dictionary.dict
Integration tips:
- cargo-fuzz uses libFuzzer backend, so all libFuzzer dict flags work
- Place dictionary file in
fuzz/directory alongside harness - Reference from harness directory:
cargo fuzz run target -- -dict=../dictionary.dict
go-fuzz (Go)
go-fuzz does not have built-in dictionary support, but you can manually seed the corpus with dictionary entries:
# Convert dictionary to corpus files
grep -o '".*"' dict.txt | while read line; do
echo -n "$line" | base64 > corpus/$(echo "$line" | md5sum | cut -d' ' -f1)
done
go-fuzz -bin=./target-fuzz.zip -workdir=.
Troubleshooting
| Issue | Cause | Solution |
|---|---|---|
| Dictionary file not loaded | Wrong path or format error | Check fuzzer output for dict parsing errors; verify file format |
| No coverage improvement | Dictionary tokens not relevant | Analyze target code for actual keywords; try different generation method |
| Syntax errors in dict file | Unescaped quotes or invalid escapes | Use \\ for backslash, \" for quotes; validate with test run |
| Fuzzer ignores long entries | Entries exceed -max_len | Keep entries under max input length, or increase -max_len |
| Too many entries slow fuzzer | Dictionary too large | Prune to 50-200 most relevant entries |
Related Skills
Tools That Use This Technique
| Skill | How It Applies |
|---|---|
| libfuzzer | Native dictionary support via -dict= flag |
| aflpp | Native dictionary support via -x flag; auto-generation with AUTODICTIONARIES |
| cargo-fuzz | Uses libFuzzer backend, inherits -dict= support |
Related Techniques
| Skill | Relationship |
|---|---|
| fuzzing-corpus | Dictionaries complement corpus: corpus provides structure, dictionary provides keywords |
| coverage-analysis | Use coverage data to validate dictionary effectiveness |
| harness-writing | Harness structure determines which dictionary tokens are useful |
Resources
Key External Resources
AFL++ Dictionaries Pre-built dictionaries for common formats (HTML, XML, JSON, SQL, etc.). Good starting point for format-specific fuzzing.
libFuzzer Dictionary Documentation Official libFuzzer documentation on dictionary format and usage. Explains token insertion strategy and performance implications.
Additional Examples
OSS-Fuzz Dictionaries
Real-world dictionaries from Google's continuous fuzzing service. Search project directories for *.dict files to see production examples.
Overcoming Fuzzing Obstacles
Codebases often contain anti-fuzzing patterns that prevent effective coverage. Checksums, global state (like time-seeded PRNGs), and validation checks can block the fuzzer from exploring deeper code paths. This technique shows how to patch your System Under Test (SUT) to bypass these obstacles during fuzzing while preserving production behavior.
Overview
Many real-world programs were not designed with fuzzing in mind. They may:
- Verify checksums or cryptographic hashes before processing input
- Rely on global state (e.g., system time, environment variables)
- Use non-deterministic random number generators
- Perform complex validation that makes it difficult for the fuzzer to generate valid inputs
These patterns make fuzzing difficult because:
- Checksums: The fuzzer must guess correct hash values (astronomically unlikely)
- Global state: Same input produces different behavior across runs (breaks determinism)
- Complex validation: The fuzzer spends effort hitting validation failures instead of exploring deeper code
The solution is conditional compilation: modify code behavior during fuzzing builds while keeping production code unchanged.
Key Concepts
| Concept | Description |
|---|---|
| SUT Patching | Modifying System Under Test to be fuzzing-friendly |
| Conditional Compilation | Code that behaves differently based on compile-time flags |
| Fuzzing Build Mode | Special build configuration that enables fuzzing-specific patches |
| False Positives | Crashes found during fuzzing that cannot occur in production |
| Determinism | Same input always produces same behavior (critical for fuzzing) |
When to Apply
Apply this technique when:
- The fuzzer gets stuck at checksum or hash verification
- Coverage reports show large blocks of unreachable code behind validation
- Code uses time-based seeds or other non-deterministic global state
- Complex validation makes it nearly impossible to generate valid inputs
- You see the fuzzer repeatedly hitting the same validation failures
Skip this technique when:
- The obstacle can be overcome with a good seed corpus or dictionary
- The validation is simple enough for the fuzzer to learn (e.g., magic bytes)
- You're doing grammar-based or structure-aware fuzzing that handles validation
- Skipping the check would introduce too many false positives
- The code is already fuzzing-friendly
Quick Reference
| Task | C/C++ | Rust |
|---|---|---|
| Check if fuzzing build | #ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION | cfg!(fuzzing) |
| Skip check during fuzzing | #ifndef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION return -1; #endif | if !cfg!(fuzzing) { return Err(...) } |
| Common obstacles | Checksums, PRNGs, time-based logic | Checksums, PRNGs, time-based logic |
| Supported fuzzers | libFuzzer, AFL++, LibAFL, honggfuzz | cargo-fuzz, libFuzzer |
Step-by-Step
Step 1: Identify the Obstacle
Run the fuzzer and analyze coverage to find code that's unreachable. Common patterns:
- Look for checksum/hash verification before deeper processing
- Check for calls to
rand(),time(), orsrand()with system seeds - Find validation functions that reject most inputs
- Identify global state initialization that differs across runs
Tools to help:
- Coverage reports (see coverage-analysis technique)
- Profiling with
-fprofile-instr-generate - Manual code inspection of entry points
Step 2: Add Conditional Compilation
Modify the obstacle to bypass it during fuzzing builds.
C/C++ Example:
// Before: Hard obstacle
if (checksum != expected_hash) {
return -1; // Fuzzer never gets past here
}
// After: Conditional bypass
if (checksum != expected_hash) {
#ifndef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
return -1; // Only enforced in production
#endif
}
// Fuzzer can now explore code beyond this check
Rust Example:
// Before: Hard obstacle
if checksum != expected_hash {
return Err(MyError::Hash); // Fuzzer never gets past here
}
// After: Conditional bypass
if checksum != expected_hash {
if !cfg!(fuzzing) {
return Err(MyError::Hash); // Only enforced in production
}
}
// Fuzzer can now explore code beyond this check
Step 3: Verify Coverage Improvement
After patching:
- Rebuild with fuzzing instrumentation
- Run the fuzzer for a short time
- Compare coverage to the unpatched version
- Confirm new code paths are being explored
Step 4: Assess False Positive Risk
Consider whether skipping the check introduces impossible program states:
- Does code after the check assume validated properties?
- Could skipping validation cause crashes that cannot occur in production?
- Is there implicit state dependency?
If false positives are likely, consider a more targeted patch (see Common Patterns below).
Common Patterns
Pattern: Bypass Checksum Validation
Use Case: Hash/checksum blocks all fuzzer progress
Before:
uint32_t computed = hash_function(data, size);
if (computed != expected_checksum) {
return ERROR_INVALID_HASH;
}
process_data(data, size);
After:
uint32_t computed = hash_function(data, size);
if (computed != expected_checksum) {
#ifndef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
return ERROR_INVALID_HASH;
#endif
}
process_data(data, size);
False positive risk: LOW - If data processing doesn't depend on checksum correctness
Pattern: Deterministic PRNG Seeding
Use Case: Non-deterministic random state prevents reproducibility
Before:
void initialize() {
srand(time(NULL)); // Different seed each run
}
After:
void initialize() {
#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
srand(12345); // Fixed seed for fuzzing
#else
srand(time(NULL));
#endif
}
False positive risk: LOW - Fuzzer can explore all code paths with fixed seed
Pattern: Careful Validation Skip
Use Case: Validation must be skipped but downstream code has assumptions
Before (Dangerous):
#ifndef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
if (!validate_config(&config)) {
return -1; // Ensures config.x != 0
}
#endif
int32_t result = 100 / config.x; // CRASH: Division by zero in fuzzing!
After (Safe):
#ifndef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
if (!validate_config(&config)) {
return -1;
}
#else
// During fuzzing, use safe defaults for failed validation
if (!validate_config(&config)) {
config.x = 1; // Prevent division by zero
config.y = 1;
}
#endif
int32_t result = 100 / config.x; // Safe in both builds
False positive risk: MITIGATED - Provides safe defaults instead of skipping
Pattern: Bypass Complex Format Validation
Use Case: Multi-step validation makes valid input generation nearly impossible
Rust Example:
// Before: Multiple validation stages
pub fn parse_message(data: &[u8]) -> Result<Message, Error> {
validate_magic_bytes(data)?;
validate_structure(data)?;
validate_checksums(data)?;
validate_crypto_signature(data)?;
deserialize_message(data)
}
// After: Skip expensive validation during fuzzing
pub fn parse_message(data: &[u8]) -> Result<Message, Error> {
validate_magic_bytes(data)?; // Keep cheap checks
if !cfg!(fuzzing) {
validate_structure(data)?;
validate_checksums(data)?;
validate_crypto_signature(data)?;
}
deserialize_message(data)
}
False positive risk: MEDIUM - Deserialization must handle malformed data gracefully
Advanced Usage
Tips and Tricks
| Tip | Why It Helps |
|---|---|
| Keep cheap validation | Magic bytes and size checks guide fuzzer without much cost |
| Use fixed seeds for PRNGs | Makes behavior deterministic while exploring all code paths |
| Patch incrementally | Skip one obstacle at a time and measure coverage impact |
| Add defensive defaults | When skipping validation, provide safe fallback values |
| Document all patches | Future maintainers need to understand fuzzing vs. production differences |
Real-World Examples
OpenSSL: Uses FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION to modify cryptographic algorithm behavior. For example, in crypto/cmp/cmp_vfy.c, certain signature checks are relaxed during fuzzing to allow deeper exploration of certificate validation logic.
ogg crate (Rust): Uses cfg!(fuzzing) to skip checksum verification during fuzzing. This allows the fuzzer to explore audio processing code without spending effort guessing correct checksums.
Measuring Patch Effectiveness
After applying patches, quantify the improvement:
- Line coverage: Use
llvm-covorcargo-covto see new reachable lines - Basic block coverage: More fine-grained than line coverage
- Function coverage: How many more functions are now reachable?
- Corpus size: Does the fuzzer generate more diverse inputs?
Effective patches typically increase coverage by 10-50% or more.
Combining with Other Techniques
Obstacle patching works well with:
- Corpus seeding: Provide valid inputs that get past initial parsing
- Dictionaries: Help fuzzer learn magic bytes and common values
- Structure-aware fuzzing: Use protobuf or grammar definitions for complex formats
- Harness improvements: Better harness can sometimes avoid obstacles entirely
Anti-Patterns
| Anti-Pattern | Problem | Correct Approach |
|---|---|---|
| Skip all validation wholesale | Creates false positives and unstable fuzzing | Skip only specific obstacles that block coverage |
| No risk assessment | False positives waste time and hide real bugs | Analyze downstream code for assumptions |
| Forget to document patches | Future maintainers don't understand the differences | Add comments explaining why patch is safe |
| Patch without measuring | Don't know if it helped | Compare coverage before and after |
| Over-patching | Makes fuzzing build diverge too much from production | Minimize differences between builds |
Tool-Specific Guidance
libFuzzer
libFuzzer automatically defines FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION during compilation.
# C++ compilation
clang++ -g -fsanitize=fuzzer,address -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION \
harness.cc target.cc -o fuzzer
# The macro is usually defined automatically by -fsanitize=fuzzer
clang++ -g -fsanitize=fuzzer,address harness.cc target.cc -o fuzzer
Integration tips:
- The macro is defined automatically; manual definition is usually unnecessary
- Use
#ifdefto check for the macro - Combine with sanitizers to detect bugs in newly reachable code
AFL++
AFL++ also defines FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION when using its compiler wrappers.
# Compilation with AFL++ wrappers
afl-clang-fast++ -g -fsanitize=address target.cc harness.cc -o fuzzer
# The macro is defined automatically by afl-clang-fast
Integration tips:
- Use
afl-clang-fastorafl-clang-ltofor automatic macro definition - Persistent mode harnesses benefit most from obstacle patching
- Consider using
AFL_LLVM_LAF_ALLfor additional input-to-state transformations
honggfuzz
honggfuzz also supports the macro when building targets.
# Compilation
hfuzz-clang++ -g -fsanitize=address target.cc harness.cc -o fuzzer
Integration tips:
- Use
hfuzz-clangorhfuzz-clang++wrappers - The macro is available for conditional compilation
- Combine with honggfuzz's feedback-driven fuzzing
cargo-fuzz (Rust)
cargo-fuzz automatically sets the fuzzing cfg option during builds.
# Build fuzz target (cfg!(fuzzing) is automatically set)
cargo fuzz build fuzz_target_name
# Run fuzz target
cargo fuzz run fuzz_target_name
Integration tips:
- Use
cfg!(fuzzing)for runtime checks in production builds - Use
#[cfg(fuzzing)]for compile-time conditional compilation - The fuzzing cfg is only set during
cargo fuzzbuilds, not regularcargo build - Can be manually enabled with
RUSTFLAGS="--cfg fuzzing"for testing
LibAFL
LibAFL supports the C/C++ macro for targets written in C/C++.
# Compilation
clang++ -g -fsanitize=address -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION \
target.cc -c -o target.o
Integration tips:
- Define the macro manually or use compiler flags
- Works the same as with libFuzzer
- Useful when building custom LibAFL-based fuzzers
Troubleshooting
| Issue | Cause | Solution |
|---|---|---|
| Coverage doesn't improve after patching | Wrong obstacle identified | Profile execution to find actual bottleneck |
| Many false positive crashes | Downstream code has assumptions | Add defensive defaults or partial validation |
| Code compiles differently | Macro not defined in all build configs | Verify macro in all source files and dependencies |
| Fuzzer finds bugs in patched code | Patch introduced invalid states | Review patch for state invariants; consider safer approach |
| Can't reproduce production bugs | Build differences too large | Minimize patches; keep validation for state-critical checks |
Related Skills
Tools That Use This Technique
| Skill | How It Applies |
|---|---|
| libfuzzer | Defines FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION automatically |
| aflpp | Supports the macro via compiler wrappers |
| honggfuzz | Uses the macro for conditional compilation |
| cargo-fuzz | Sets cfg!(fuzzing) for Rust conditional compilation |
Related Techniques
| Skill | Relationship |
|---|---|
| fuzz-harness-writing | Better harnesses may avoid obstacles; patching enables deeper exploration |
| coverage-analysis | Use coverage to identify obstacles and measure patch effectiveness |
| corpus-seeding | Seed corpus can help overcome obstacles without patching |
| dictionary-generation | Dictionaries help with magic bytes but not checksums or complex validation |
Resources
Key External Resources
OpenSSL Fuzzing Documentation
OpenSSL's fuzzing infrastructure demonstrates large-scale use of FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION. The project uses this macro to modify cryptographic validation, certificate parsing, and other security-critical code paths to enable deeper fuzzing while maintaining production correctness.
LibFuzzer Documentation on Flags Official LLVM documentation for libFuzzer, including how the fuzzer defines compiler macros and how to use them effectively. Covers integration with sanitizers and coverage instrumentation.
Rust cfg Attribute Reference
Complete reference for Rust conditional compilation, including cfg!(fuzzing) and cfg!(test). Explains compile-time vs. runtime conditional compilation and best practices.
Genotoxic
Combines mutation testing and necessist (test statement removal) with code graph analysis to triage findings into actionable categories: false positives, missing unit tests, and fuzzing targets.
When to Use
- After mutation testing reveals survived mutants that need triage
- Identifying where unit tests would have the highest impact
- Finding functions that need fuzz harnesses instead of unit tests
- Prioritizing test improvements using data flow context
- Filtering out harmless mutants from actionable ones
- Finding unnecessary test statements that indicate weak assertions (necessist)
When NOT to Use
- Codebase has no existing test suite (write tests first)
- Pure documentation or configuration changes
- Single-file scripts with trivial logic
Prerequisites
- trailmark installed — if
uv run trailmarkfails, run:uv tool install trailmark
Python snippets: uv run --with trailmark python - (a tool env is not importable)
**DO NOT** fall back to "manual verification" or "manual analysis"
as a substitute for running trailmark. Install it first. If installation
fails, report the error instead of switching to manual analysis.
- A **mutation testing framework** for the target language — if the framework
command fails (not found, not installed), install it using the instructions
in [references/mutation-frameworks.md](references/mutation-frameworks.md).
**DO NOT** fall back to "manual mutation analysis" or skip mutation testing.
Install the framework first. If installation fails, report the error
instead of switching to manual mutation analysis.
- **necessist** (optional, recommended) — if the target language is
supported (Go, Rust, Solidity/Foundry, TypeScript/Hardhat,
TypeScript/Vitest, Rust/Anchor), install with `cargo install necessist`.
See [references/mutation-frameworks.md](references/mutation-frameworks.md)
for details.
- An existing test suite that passes
- **macOS environment**: Run `ulimit -n 1024` before any `mull-runner`
invocation. macOS Tahoe (26+) sets unlimited file descriptors by
default, which crashes Mull's subprocess spawning. See
[references/mutation-frameworks.md](references/mutation-frameworks.md)
for details.
---
## Rationalizations to Reject
| Rationalization | Why It's Wrong | Required Action |
|-----------------|----------------|-----------------|
| "All survived mutants need tests" | Many are harmless or equivalent | Triage before writing tests |
| "Mutation testing is too noisy" | Noise means you're not triaging | Use graph data to filter |
| "Unit tests cover everything" | Complex data flows need fuzzing | Check entrypoint reachability |
| "Dead code mutants don't matter" | Dead code should be removed | Flag for cleanup |
| "Low complexity = low risk" | Boundary bugs hide in simple code | Check mutant location |
| "Tool isn't installed, I'll do it manually" | Manual analysis misses what tooling catches | Install the tool first |
| "Necessist isn't mutation testing, skip it" | Necessist finds what mutation testing misses: weak tests | Run both when the language supports it |
---
## Quick Start
```bash
# 1. Build the code graph
uv run trailmark analyze --language auto --summary {targetDir}
# 2. Run mutation testing (language-dependent)
# Python:
uv run mutmut run --paths-to-mutate {targetDir}/src
uv run mutmut results
# 2b. Run necessist (if language supported)
necessist
# 3. Analyze results with this skill's workflow (Phase 3)
Workflow Overview
Phase 1: Graph Build → Parse codebase with trailmark
↓
Phase 2: Mutation Run → Execute mutation testing framework
Phase 2b: Necessist Run → Remove test statements (optional, parallel)
↓
Phase 3: Triage → Classify findings using graph data
↓
Output: Categorized Report
├── Corroborated (both tools flag same function — highest value)
├── False Positives (harmless, skip)
├── Missing Tests (write unit tests)
└── Fuzzing Targets (set up fuzz harnesses)
Decision Tree
├─ Need to set up mutation testing for a language?
│ └─ Read: references/mutation-frameworks.md
│
├─ Need to set up necessist or find weak test statements?
│ └─ Read: references/mutation-frameworks.md (Necessist section)
│
├─ Need to understand the triage criteria in depth?
│ └─ Read: references/triage-methodology.md
│
├─ Need to understand how graph data informs triage?
│ └─ Read: references/graph-analysis.md
│
└─ Already have results + graph? Use Phase 3 below.
Phase 1: Build Code Graph and Run Pre-Analysis
Parse the target codebase with trailmark and run pre-analysis before mutation testing. Pre-analysis computes blast radius, entry points, privilege boundaries, and taint propagation, which Phase 3 uses for triage.
uv run trailmark analyze --language auto --summary {targetDir}
Use the QueryEngine API to build the graph and run pre-analysis:
QueryEngine.from_directory("{targetDir}", language="auto")- Call
engine.preanalysis()— mandatory before triage - Export with
engine.to_json()for cross-referencing with mutation results
If auto-detection is wrong for the target, rerun with an explicit language or
comma-separated list such as python,rust.
See references/graph-analysis.md for the full API: node mapping, reachability queries, blast radius, and pre-analysis subgraph lookups.
Phase 2: Run Mutation Testing
Select and run the appropriate framework. See references/mutation-frameworks.md for language-specific setup.
Capture survived mutants. Each framework reports differently, but extract these fields per mutant:
| Field | Description |
|---|---|
| File path | Source file containing the mutant |
| Line number | Line where mutation was applied |
| Mutation type | What was changed (operator, value, etc.) |
| Status | survived, killed, timeout, error |
Filter to survived mutants only for Phase 3.
Phase 2b: Run Necessist (Optional)
If the target language is supported (Go, Rust, Solidity/Foundry, TypeScript/Hardhat, TypeScript/Vitest, Rust/Anchor), run necessist to find unnecessary test statements. This runs independently of Phase 2 and can execute in parallel.
# Auto-detect framework
necessist
# Or target specific test files
necessist tests/test_parser.rs
# Export results
necessist --dump
Filter to findings where the test passed after removal. See references/mutation-frameworks.md for framework-specific configuration and the normalized record format.
Map each removal to a production function using the algorithm in references/graph-analysis.md.
Phase 3: Triage Findings
For each survived mutant and each necessist removal, determine its triage bucket using graph data. Necessist removals must first be mapped to a production function (see references/graph-analysis.md).
Quick Classification (Mutation Testing)
| Signal | Bucket | Reasoning |
|---|---|---|
| No callers in graph | False Positive | Dead code, mutant is unreachable |
| Only test callers | False Positive | Test infrastructure, not production |
| Logging/display string | False Positive | Cosmetic, no behavioral impact |
| Equivalent mutant | False Positive | Behavior unchanged despite mutation |
| Simple function, low CC, no entrypoint path | Missing Tests | Unit test is straightforward |
| Error handling path | Missing Tests | Should have negative test cases |
| Boundary condition (off-by-one) | Missing Tests | Property-based test candidate |
| Pure function, deterministic | Missing Tests | Easy to test, high value |
| High CC (>10), entrypoint reachable | Fuzzing Target | Complex + exposed = fuzz it |
| Parser/validator/deserializer | Fuzzing Target | Structured input handling |
| Many callers (>10) + moderate CC | Fuzzing Target | High blast radius |
| Binary/wire protocol handling | Fuzzing Target | Fuzzers excel at format testing |
Quick Classification (Necessist)
| Signal | Bucket | Reasoning |
|---|---|---|
| Redundant setup or debug call | False Positive | Statement genuinely unnecessary |
| Cannot map to production function | False Positive | No graph context for triage |
| Call removed, no assertion checks its effect | Missing Tests | Test has weak assertions |
| Assertion removed, test still passes | Missing Tests | Redundant or insufficient coverage |
| Maps to high-CC entrypoint-reachable function | Fuzzing Target | Complex + exposed + weak test |
When both mutation testing and necessist flag the same production function, mark as corroborated — highest confidence finding.
For detailed criteria, see references/triage-methodology.md.
Graph Queries for Triage
For each mutant, map it to its containing graph node and use pre-analysis subgraphs (tainted, high_blast_radius, privilege_boundary) from Phase 1 to classify it. The classification logic checks: no callers → false positive, privilege boundary → fuzzing, high CC + tainted → fuzzing, high blast radius → fuzzing, otherwise → missing tests.
See references/graph-analysis.md for
the batch_triage implementation and node mapping functions.
Output Format
Generate a markdown report:
# Genotoxic Triage Report
## Summary
- Total survived mutants: N
- Total necessist removals: N
- Corroborated findings: N
- False positives: N (N%)
- Missing test coverage: N (N%)
- Fuzzing targets: N (N%)
## Corroborated Findings
| File | Line | Function | Mutation Signal | Necessist Signal | Action |
|------|------|----------|----------------|------------------|--------|
## False Positives
| File | Line | Mutation | Reason | Source |
|------|------|----------|--------|--------|
## Missing Test Coverage
| File | Line | Function | CC | Callers | Suggested Test | Source |
|------|------|----------|----|---------|----------------|--------|
## Fuzzing Targets
| File | Line | Function | CC | Entrypoint Path | Blast Radius | Source |
|------|------|----------|----|-----------------|--------------|--------|
The Source column is mutation, necessist, or corroborated.
Write the report to GENOTOXIC_REPORT.md in the working directory.
Quality Checklist
Before delivering:
- Trailmark graph built for target language
- Mutation framework ran to completion
- Necessist ran (if language supported) or noted as not applicable
- All survived mutants triaged (none unclassified)
- All necessist removals triaged (if applicable)
- Corroborated findings identified (if both tools ran)
- False positives have clear justifications
- Missing test items include suggested test type
- Fuzzing targets include entrypoint paths and blast radius
- Report file written to
GENOTOXIC_REPORT.md - User notified with summary statistics
Integration
trailmark skill:
- Phase 1: Build code graph, query complexity and entrypoints
- Phase 3: Caller analysis, reachability, blast radius
property-based-testing skill:
- Missing test coverage items involving boundary conditions
- Roundtrip/idempotence properties for serialization mutants
testing-handbook-skills (fuzzing):
- Fuzzing target items: use
harness-writing,cargo-fuzz,atheris
Supporting Documentation
- references/mutation-frameworks.md - Language-specific framework setup, output parsing, and necessist configuration
- references/triage-methodology.md - Detailed triage criteria, edge cases, and worked examples for both mutation testing and necessist
- references/graph-analysis.md - Graph query patterns, test-to-production mapping, and result merging
First-time users: Start with Phase 1 (graph build), then run mutations, then use the Quick Classification table in Phase 3.
Experienced users: Jump to Phase 3 and use the Decision Tree to load specific reference material.
Writing Fuzzing Harnesses
A fuzzing harness is the entrypoint function that receives random data from the fuzzer and routes it to your system under test (SUT). The quality of your harness directly determines which code paths get exercised and whether critical bugs are found. A poorly written harness can miss entire subsystems or produce non-reproducible crashes.
Overview
The harness is the bridge between the fuzzer's random byte generation and your application's API. It must parse raw bytes into meaningful inputs, call target functions, and handle edge cases gracefully. The most important part of any fuzzing setup is the harness—if written poorly, critical parts of your application may not be covered.
Key Concepts
| Concept | Description |
|---|---|
| Harness | Function that receives fuzzer input and calls target code under test |
| SUT | System Under Test—the code being fuzzed |
| Entry point | Function signature required by the fuzzer (e.g., LLVMFuzzerTestOneInput) |
| FuzzedDataProvider | Helper class for structured extraction of typed data from raw bytes |
| Determinism | Property that ensures same input always produces same behavior |
| Interleaved fuzzing | Single harness that exercises multiple operations based on input |
When to Apply
Apply this technique when:
- Creating a new fuzz target for the first time
- Fuzz campaign has low code coverage or isn't finding bugs
- Crashes found during fuzzing are not reproducible
- Target API requires complex or structured inputs
- Multiple related functions should be tested together
Skip this technique when:
- Using existing well-tested harnesses from your project
- Tool provides automatic harness generation that meets your needs
- Target already has comprehensive fuzzing infrastructure
Quick Reference
| Task | Pattern |
|---|---|
| Minimal C++ harness | extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) |
| Minimal Rust harness | `fuzz_target!( |
| Size validation | if (size < MIN_SIZE) return 0; |
| Cast to integers | uint32_t val = *(uint32_t*)(data); |
| Use FuzzedDataProvider | FuzzedDataProvider fuzzed_data(data, size); |
| Extract typed data (C++) | auto val = fuzzed_data.ConsumeIntegral<uint32_t>(); |
| Extract string (C++) | auto str = fuzzed_data.ConsumeBytesWithTerminator<char>(32, 0xFF); |
Step-by-Step
Step 1: Identify Entry Points
Find functions in your codebase that:
- Accept external input (parsers, validators, protocol handlers)
- Parse complex data formats (JSON, XML, binary protocols)
- Perform security-critical operations (authentication, cryptography)
- Have high cyclomatic complexity or many branches
Good targets are typically:
- Protocol parsers
- File format parsers
- Serialization/deserialization functions
- Input validation routines
Step 2: Write Minimal Harness
Start with the simplest possible harness that calls your target function:
C/C++:
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
target_function(data, size);
return 0;
}
Rust:
#![no_main]
use libfuzzer_sys::fuzz_target;
fuzz_target!(|data: &[u8]| {
target_function(data);
});
Step 3: Add Input Validation
Reject inputs that are too small or too large to be meaningful:
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
// Ensure minimum size for meaningful input
if (size < MIN_INPUT_SIZE || size > MAX_INPUT_SIZE) {
return 0;
}
target_function(data, size);
return 0;
}
Rationale: The fuzzer generates random inputs of all sizes. Your harness must handle empty, tiny, huge, or malformed inputs without causing unexpected issues in the harness itself (crashes in the SUT are fine—that's what we're looking for).
Step 4: Structure the Input
For APIs that require typed data (integers, strings, etc.), use casting or helpers like FuzzedDataProvider:
Simple casting:
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
if (size != 2 * sizeof(uint32_t)) {
return 0;
}
uint32_t numerator = *(uint32_t*)(data);
uint32_t denominator = *(uint32_t*)(data + sizeof(uint32_t));
divide(numerator, denominator);
return 0;
}
Using FuzzedDataProvider:
#include "FuzzedDataProvider.h"
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
FuzzedDataProvider fuzzed_data(data, size);
size_t allocation_size = fuzzed_data.ConsumeIntegral<size_t>();
std::vector<char> str1 = fuzzed_data.ConsumeBytesWithTerminator<char>(32, 0xFF);
std::vector<char> str2 = fuzzed_data.ConsumeBytesWithTerminator<char>(32, 0xFF);
concat(&str1[0], str1.size(), &str2[0], str2.size(), allocation_size);
return 0;
}
Step 5: Test and Iterate
Run the fuzzer and monitor:
- Code coverage (are all interesting paths reached?)
- Executions per second (is it fast enough?)
- Crash reproducibility (can you reproduce crashes with saved inputs?)
Iterate on the harness to improve these metrics.
Common Patterns
Pattern: Beyond Byte Arrays—Casting to Integers
Use Case: When target expects primitive types like integers or floats
Implementation:
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
// Ensure exactly 2 4-byte numbers
if (size != 2 * sizeof(uint32_t)) {
return 0;
}
// Split input into two integers
uint32_t numerator = *(uint32_t*)(data);
uint32_t denominator = *(uint32_t*)(data + sizeof(uint32_t));
divide(numerator, denominator);
return 0;
}
Rust equivalent:
fuzz_target!(|data: &[u8]| {
if data.len() != 2 * std::mem::size_of::<i32>() {
return;
}
let numerator = i32::from_ne_bytes([data[0], data[1], data[2], data[3]]);
let denominator = i32::from_ne_bytes([data[4], data[5], data[6], data[7]]);
divide(numerator, denominator);
});
Why it works: Any 8-byte input is valid. The fuzzer learns that inputs must be exactly 8 bytes, and every bit flip produces a new, potentially interesting input.
Pattern: FuzzedDataProvider for Complex Inputs
Use Case: When target requires multiple strings, integers, or variable-length data
Implementation:
#include "FuzzedDataProvider.h"
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
FuzzedDataProvider fuzzed_data(data, size);
// Extract different types of data
size_t allocation_size = fuzzed_data.ConsumeIntegral<size_t>();
// Consume variable-length strings with terminator
std::vector<char> str1 = fuzzed_data.ConsumeBytesWithTerminator<char>(32, 0xFF);
std::vector<char> str2 = fuzzed_data.ConsumeBytesWithTerminator<char>(32, 0xFF);
char* result = concat(&str1[0], str1.size(), &str2[0], str2.size(), allocation_size);
if (result != NULL) {
free(result);
}
return 0;
}
Why it helps: FuzzedDataProvider handles the complexity of extracting structured data from a byte stream. It's particularly useful for APIs that need multiple parameters of different types.
Pattern: Interleaved Fuzzing
Use Case: When multiple related operations should be tested in a single harness
Implementation:
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
if (size < 1 + 2 * sizeof(int32_t)) {
return 0;
}
// First byte selects operation
uint8_t mode = data[0];
// Next bytes are operands
int32_t numbers[2];
memcpy(numbers, data + 1, 2 * sizeof(int32_t));
int32_t result = 0;
switch (mode % 4) {
case 0:
result = add(numbers[0], numbers[1]);
break;
case 1:
result = subtract(numbers[0], numbers[1]);
break;
case 2:
result = multiply(numbers[0], numbers[1]);
break;
case 3:
result = divide(numbers[0], numbers[1]);
break;
}
// Prevent compiler from optimizing away the calls
printf("%d", result);
return 0;
}
Advantages:
- Faster to write one harness than multiple individual harnesses
- Single shared corpus means interesting inputs for one operation may be interesting for others
- Can discover bugs in interactions between operations
When to use:
- Operations share similar input types
- Operations are logically related (e.g., arithmetic operations, CRUD operations)
- Single corpus makes sense across all operations
Pattern: Structure-Aware Fuzzing with Arbitrary (Rust)
Use Case: When fuzzing Rust code that uses custom structs
Implementation:
use arbitrary::Arbitrary;
#[derive(Debug, Arbitrary)]
pub struct Name {
data: String
}
impl Name {
pub fn check_buf(&self) {
let data = self.data.as_bytes();
if data.len() > 0 && data[0] == b'a' {
if data.len() > 1 && data[1] == b'b' {
if data.len() > 2 && data[2] == b'c' {
process::abort();
}
}
}
}
}
Harness with arbitrary:
#![no_main]
use libfuzzer_sys::fuzz_target;
fuzz_target!(|data: your_project::Name| {
data.check_buf();
});
Add to Cargo.toml:
[dependencies]
arbitrary = { version = "1", features = ["derive"] }
Why it helps: The arbitrary crate automatically handles deserialization of raw bytes into your Rust structs, reducing boilerplate and ensuring valid struct construction.
Limitation: The arbitrary crate doesn't offer reverse serialization, so you can't manually construct byte arrays that map to specific structs. This works best when starting from an empty corpus (fine for libFuzzer, problematic for AFL++).
Advanced Usage
Tips and Tricks
| Tip | Why It Helps |
|---|---|
| Start with parsers | High bug density, clear entry points, easy to harness |
| Mock I/O operations | Prevents hangs from blocking I/O, enables determinism |
| Use FuzzedDataProvider | Simplifies extraction of structured data from raw bytes |
| Reset global state | Ensures each iteration is independent and reproducible |
| Free resources in harness | Prevents memory exhaustion during long campaigns |
| Avoid logging in harness | Logging is slow—fuzzing needs 100s-1000s exec/sec |
| Test harness manually first | Run harness with known inputs before starting campaign |
| Check coverage early | Ensure harness reaches expected code paths |
Structure-Aware Fuzzing with Protocol Buffers
For highly structured input formats, consider using Protocol Buffers as an intermediate format with custom mutators:
// Define your input format in .proto file
// Use libprotobuf-mutator to generate valid mutations
// This ensures fuzzer mutates message contents, not the protobuf encoding itself
This approach is more setup but prevents the fuzzer from wasting time on unparseable inputs. See structure-aware fuzzing documentation for details.
Handling Non-Determinism
Problem: Random values or timing dependencies cause non-reproducible crashes.
Solutions:
- Replace
rand()with deterministic PRNG seeded from fuzzer input:uint32_t seed = fuzzed_data.ConsumeIntegral<uint32_t>(); srand(seed); - Mock system calls that return time, PIDs, or random data
- Avoid reading from
/dev/randomor/dev/urandom
Resetting Global State
If your SUT uses global state (singletons, static variables), reset it between iterations:
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
// Reset global state before each iteration
global_reset();
target_function(data, size);
// Clean up resources
global_cleanup();
return 0;
}
Rationale: Global state can cause crashes after N iterations rather than on a specific input, making bugs non-reproducible.
Practical Harness Rules
Follow these rules to ensure effective fuzzing harnesses:
| Rule | Rationale |
|---|---|
| Handle all input sizes | Fuzzer generates empty, tiny, huge inputs—harness must handle gracefully |
Never call exit() | Calling exit() stops the fuzzer process. Use abort() in SUT if needed |
| Join all threads | Each iteration must run to completion before next iteration starts |
| Be fast | Aim for 100s-1000s executions/sec. Avoid logging, high complexity, excess memory |
| Maintain determinism | Same input must always produce same behavior for reproducibility |
| Avoid global state | Global state reduces reproducibility—reset between iterations if unavoidable |
| Use narrow targets | Don't fuzz PNG and TCP in same harness—different formats need separate targets |
| Free resources | Prevent memory leaks that cause resource exhaustion during long campaigns |
Note: These guidelines apply not just to harness code, but to the entire SUT. If the SUT violates these rules, consider patching it (see the fuzzing obstacles technique).
Anti-Patterns
| Anti-Pattern | Problem | Correct Approach |
|---|---|---|
| Global state without reset | Non-deterministic crashes | Reset all globals at start of harness |
| Blocking I/O or network calls | Hangs fuzzer, wastes time | Mock I/O, use in-memory buffers |
| Memory leaks in harness | Resource exhaustion kills campaign | Free all allocations before returning |
Calling exit() in SUT | Stops entire fuzzing process | Use abort() or return error codes |
| Heavy logging in harness | Reduces exec/sec by orders of magnitude | Disable logging during fuzzing |
| Too many operations per iteration | Slows down fuzzer | Keep iterations fast and focused |
| Mixing unrelated input formats | Corpus entries not useful across formats | Separate harnesses for different formats |
| Not validating input size | Harness crashes on edge cases | Check size before accessing data |
Tool-Specific Guidance
libFuzzer
Harness signature:
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
// Your code here
return 0; // Non-zero return is reserved for future use
}
Compilation:
clang++ -fsanitize=fuzzer,address -g harness.cc -o fuzz_target
Integration tips:
- Use
FuzzedDataProvider.hfor structured input extraction - Compile with
-fsanitize=fuzzerto link the fuzzing runtime - Add sanitizers (
-fsanitize=address,undefined) to detect more bugs - Use
-gfor better stack traces when crashes occur - libFuzzer can start with empty corpus—no seed inputs required
Running:
./fuzz_target corpus_dir/
Resources:
AFL++
AFL++ supports multiple harness styles. For best performance, use persistent mode:
Persistent mode harness:
#include <unistd.h>
int main(int argc, char **argv) {
#ifdef __AFL_HAVE_MANUAL_CONTROL
__AFL_INIT();
#endif
unsigned char buf[MAX_SIZE];
while (__AFL_LOOP(10000)) {
// Read input from stdin
ssize_t len = read(0, buf, sizeof(buf));
if (len <= 0) break;
// Call target function
target_function(buf, len);
}
return 0;
}
Compilation:
afl-clang-fast++ -g harness.cc -o fuzz_target
Integration tips:
- Use persistent mode (
__AFL_LOOP) for 10-100x speedup - Consider deferred initialization (
__AFL_INIT()) to skip setup overhead - AFL++ requires at least one seed input in the corpus directory
- Use
AFL_USE_ASAN=1orAFL_USE_UBSAN=1for sanitizer builds
Running:
afl-fuzz -i seeds/ -o findings/ -- ./fuzz_target
cargo-fuzz (Rust)
Harness signature:
#![no_main]
use libfuzzer_sys::fuzz_target;
fuzz_target!(|data: &[u8]| {
// Your code here
});
With structured input (arbitrary crate):
#![no_main]
use libfuzzer_sys::fuzz_target;
fuzz_target!(|data: YourStruct| {
data.check();
});
Creating harness:
cargo fuzz init
cargo fuzz add my_target
Integration tips:
- Use
arbitrarycrate for automatic struct deserialization - cargo-fuzz wraps libFuzzer, so all libFuzzer features work
- Compile with sanitizers automatically via cargo-fuzz
- Harnesses go in
fuzz/fuzz_targets/directory
Running:
cargo +nightly fuzz run my_target
Resources:
go-fuzz
Harness signature:
// +build gofuzz
package mypackage
func Fuzz(data []byte) int {
// Call target function
target(data)
// Return codes:
// -1 if input is invalid
// 0 if input is valid but not interesting
// 1 if input is interesting (e.g., added new coverage)
return 0
}
Building:
go-fuzz-build
Integration tips:
- Return 1 for inputs that add coverage (optional—fuzzer can detect automatically)
- Return -1 for invalid inputs to deprioritize similar mutations
- go-fuzz handles persistence automatically
Running:
go-fuzz -bin=./mypackage-fuzz.zip -workdir=fuzz
Troubleshooting
| Issue | Cause | Solution |
|---|---|---|
| Low executions/sec | Harness is too slow (logging, I/O, complexity) | Profile harness, remove bottlenecks, mock I/O |
| No crashes found | Coverage not reaching buggy code | Check coverage, improve harness to reach more paths |
| Non-reproducible crashes | Non-determinism or global state | Remove randomness, reset globals between iterations |
| Fuzzer exits immediately | Harness calls exit() | Replace exit() with abort() or return error |
| Out of memory errors | Memory leaks in harness or SUT | Free allocations, use leak sanitizer to find leaks |
| Crashes on empty input | Harness doesn't validate size | Add if (size < MIN_SIZE) return 0; |
| Corpus not growing | Inputs too constrained or format too strict | Use FuzzedDataProvider or structure-aware fuzzing |
Related Skills
Tools That Use This Technique
| Skill | How It Applies |
|---|---|
| libfuzzer | Uses LLVMFuzzerTestOneInput harness signature with FuzzedDataProvider |
| aflpp | Supports persistent mode harnesses with __AFL_LOOP for performance |
| cargo-fuzz | Uses Rust-specific fuzz_target! macro with arbitrary crate integration |
| atheris | Python harness takes bytes, calls Python functions |
| ossfuzz | Requires harnesses in specific directory structure for cloud fuzzing |
Related Techniques
| Skill | Relationship |
|---|---|
| coverage-analysis | Measure harness effectiveness—are you reaching target code? |
| address-sanitizer | Detects bugs found by harness (buffer overflows, use-after-free) |
| fuzzing-dictionary | Provide tokens to help fuzzer pass format checks in harness |
| fuzzing-obstacles | Patch SUT when it violates harness rules (exit, non-determinism) |
Resources
Key External Resources
Split Inputs in libFuzzer - Google Fuzzing Docs Explains techniques for handling multiple input parameters in a single fuzzing harness, including use of magic separators and FuzzedDataProvider.
Structure-Aware Fuzzing with Protocol Buffers Advanced technique using protobuf as intermediate format with custom mutators to ensure fuzzer mutates message contents rather than format encoding.
libFuzzer Documentation Official LLVM documentation covering harness requirements, best practices, and advanced features.
cargo-fuzz Book Comprehensive guide to writing Rust fuzzing harnesses with cargo-fuzz and the arbitrary crate.
Video Resources
- Effective File Format Fuzzing - Conference talk on writing harnesses for file format parsers
- Modern Fuzzing of C/C++ Projects - Tutorial covering harness design patterns
LibAFL
LibAFL is a modular fuzzing library that implements features from AFL-based fuzzers like AFL++. Unlike traditional fuzzers, LibAFL provides all functionality in a modular and customizable way as a Rust library. It can be used as a drop-in replacement for libFuzzer or as a library to build custom fuzzers from scratch.
When to Use
| Fuzzer | Best For | Complexity |
|---|---|---|
| libFuzzer | Quick setup, single-threaded | Low |
| AFL++ | Multi-core, general purpose | Medium |
| LibAFL | Custom fuzzers, advanced features, research | High |
Choose LibAFL when:
- You need custom mutation strategies or feedback mechanisms
- Standard fuzzers don't support your target architecture
- You want to implement novel fuzzing techniques
- You need fine-grained control over fuzzing components
- You're conducting fuzzing research
Quick Start
LibAFL can be used as a drop-in replacement for libFuzzer with minimal setup:
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
// Call your code with fuzzer-provided data
my_function(data, size);
return 0;
}
Build LibAFL's libFuzzer compatibility layer:
git clone https://github.com/AFLplusplus/LibAFL
cd LibAFL/libafl_libfuzzer_runtime
./build.sh
Compile and run:
clang++ -DNO_MAIN -g -O2 -fsanitize=fuzzer-no-link libFuzzer.a harness.cc main.cc -o fuzz
./fuzz corpus/
Installation
Prerequisites
- Clang/LLVM 15-18
- Rust (via rustup)
- Additional system dependencies
Linux/macOS
Install Clang:
apt install clang
Or install a specific version via apt.llvm.org:
wget https://apt.llvm.org/llvm.sh
chmod +x llvm.sh
sudo ./llvm.sh 15
Configure environment for Rust:
export RUSTFLAGS="-C linker=/usr/bin/clang-15"
export CC="clang-15"
export CXX="clang++-15"
Install Rust:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
Install additional dependencies:
apt install libssl-dev pkg-config
For libFuzzer compatibility mode, install nightly Rust:
rustup toolchain install nightly --component llvm-tools
Verification
Build LibAFL to verify installation:
cd LibAFL/libafl_libfuzzer_runtime
./build.sh
# Should produce libFuzzer.a
Writing a Harness
LibAFL harnesses follow the same pattern as libFuzzer when using drop-in replacement mode:
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
// Your fuzzing target code here
return 0;
}
When building custom fuzzers with LibAFL as a Rust library, harness logic is integrated directly into the fuzzer. See the "Writing a Custom Fuzzer" section below for the full pattern.
See Also: For detailed harness writing techniques, see the harness-writing technique skill.
Usage Modes
LibAFL supports two primary usage modes:
1. libFuzzer Drop-in Replacement
Use LibAFL as a replacement for libFuzzer with existing harnesses.
Compilation:
clang++ -DNO_MAIN -g -O2 -fsanitize=fuzzer-no-link libFuzzer.a harness.cc main.cc -o fuzz
Running:
./fuzz corpus/
Recommended for long campaigns:
./fuzz -fork=1 -ignore_crashes=1 corpus/
2. Custom Fuzzer as Rust Library
Build a fully customized fuzzer using LibAFL components.
Create project:
cargo init --lib my_fuzzer
cd my_fuzzer
cargo add libafl@0.13 libafl_targets@0.13 libafl_bolts@0.13 libafl_cc@0.13 \
--features "libafl_targets@0.13/libfuzzer,libafl_targets@0.13/sancov_pcguard_hitcounts"
Configure Cargo.toml:
[lib]
crate-type = ["staticlib"]
Writing a Custom Fuzzer
See Also: For detailed harness writing techniques, patterns for handling complex inputs, and advanced strategies, see the fuzz-harness-writing technique skill.
Fuzzer Components
A LibAFL fuzzer consists of modular components:
- Observers - Collect execution feedback (coverage, timing)
- Feedback - Determine if inputs are interesting
- Objective - Define fuzzing goals (crashes, timeouts)
- State - Maintain corpus and metadata
- Mutators - Generate new inputs
- Scheduler - Select which inputs to mutate
- Executor - Run the target with inputs
Basic Fuzzer Structure
use libafl::prelude::*;
use libafl_bolts::prelude::*;
use libafl_targets::{libfuzzer_test_one_input, std_edges_map_observer};
#[no_mangle]
pub extern "C" fn libafl_main() {
let mut run_client = |state: Option<_>, mut restarting_mgr, _core_id| {
// 1. Setup observers
let edges_observer = HitcountsMapObserver::new(
unsafe { std_edges_map_observer("edges") }
).track_indices();
let time_observer = TimeObserver::new("time");
// 2. Define feedback
let mut feedback = feedback_or!(
MaxMapFeedback::new(&edges_observer),
TimeFeedback::new(&time_observer)
);
// 3. Define objective
let mut objective = feedback_or_fast!(
CrashFeedback::new(),
TimeoutFeedback::new()
);
// 4. Create or restore state
let mut state = state.unwrap_or_else(|| {
StdState::new(
StdRand::new(),
InMemoryCorpus::new(),
OnDiskCorpus::new(&output_dir).unwrap(),
&mut feedback,
&mut objective,
).unwrap()
});
// 5. Setup mutator
let mutator = StdScheduledMutator::new(havoc_mutations());
let mut stages = tuple_list!(StdMutationalStage::new(mutator));
// 6. Setup scheduler
let scheduler = IndexesLenTimeMinimizerScheduler::new(
&edges_observer,
QueueScheduler::new()
);
// 7. Create fuzzer
let mut fuzzer = StdFuzzer::new(scheduler, feedback, objective);
// 8. Define harness
let mut harness = |input: &BytesInput| {
let buf = input.target_bytes().as_slice();
libfuzzer_test_one_input(buf);
ExitKind::Ok
};
// 9. Setup executor
let mut executor = InProcessExecutor::with_timeout(
&mut harness,
tuple_list!(edges_observer, time_observer),
&mut fuzzer,
&mut state,
&mut restarting_mgr,
timeout,
)?;
// 10. Load initial inputs
if state.must_load_initial_inputs() {
state.load_initial_inputs(
&mut fuzzer,
&mut executor,
&mut restarting_mgr,
&input_dir
)?;
}
// 11. Start fuzzing
fuzzer.fuzz_loop(&mut stages, &mut executor, &mut state, &mut restarting_mgr)?;
Ok(())
};
// Launch fuzzer
Launcher::builder()
.run_client(&mut run_client)
.cores(&cores)
.build()
.launch()
.unwrap();
}
Compilation
Verbose Mode
Manually specify all instrumentation flags:
clang++-15 -DNO_MAIN -g -O2 \
-fsanitize-coverage=trace-pc-guard \
-fsanitize=address \
-Wl,--whole-archive target/release/libmy_fuzzer.a -Wl,--no-whole-archive \
main.cc harness.cc -o fuzz
Compiler Wrapper (Recommended)
Create a LibAFL compiler wrapper to handle instrumentation automatically.
Create src/bin/libafl_cc.rs:
use libafl_cc::{ClangWrapper, CompilerWrapper, Configuration, ToolWrapper};
pub fn main() {
let args: Vec<String> = env::args().collect();
let mut cc = ClangWrapper::new();
cc.cpp(is_cpp)
.parse_args(&args)
.link_staticlib(&dir, "my_fuzzer")
.add_args(&Configuration::GenerateCoverageMap.to_flags().unwrap())
.add_args(&Configuration::AddressSanitizer.to_flags().unwrap())
.run()
.unwrap();
}
Compile and use:
cargo build --release
target/release/libafl_cxx -DNO_MAIN -g -O2 main.cc harness.cc -o fuzz
See Also: For detailed sanitizer configuration, common issues, and advanced flags, see the address-sanitizer and undefined-behavior-sanitizer technique skills.
Running Campaigns
Basic Run
./fuzz --cores 0 --input corpus/
Multi-Core Fuzzing
./fuzz --cores 0,8-15 --input corpus/
This runs 9 clients: one on core 0, and 8 on cores 8-15.
With Options
./fuzz --cores 0-7 --input corpus/ --output crashes/ --timeout 1000
Text User Interface (TUI)
Enable graphical statistics view:
./fuzz -tui=1 corpus/
Interpreting Output
| Output | Meaning |
|---|---|
corpus: N | Number of interesting test cases found |
objectives: N | Number of crashes/timeouts found |
executions: N | Total number of target invocations |
exec/sec: N | Current execution throughput |
edges: X% | Code coverage percentage |
clients: N | Number of parallel fuzzing processes |
The fuzzer emits two main event types:
- UserStats - Regular heartbeat with current statistics
- Testcase - New interesting input discovered
Advanced Usage
Tips and Tricks
| Tip | Why It Helps |
|---|---|
Use -fork=1 -ignore_crashes=1 | Continue fuzzing after first crash |
Use InMemoryOnDiskCorpus | Persist corpus across restarts |
Enable TUI with -tui=1 | Better visualization of progress |
| Use specific LLVM version | Avoid compatibility issues |
Set RUSTFLAGS correctly | Prevent linking errors |
Crash Deduplication
Avoid storing duplicate crashes from the same bug:
Add backtrace observer:
let backtrace_observer = BacktraceObserver::owned(
"BacktraceObserver",
libafl::observers::HarnessType::InProcess
);
Update executor:
let mut executor = InProcessExecutor::with_timeout(
&mut harness,
tuple_list!(edges_observer, time_observer, backtrace_observer),
&mut fuzzer,
&mut state,
&mut restarting_mgr,
timeout,
)?;
Update objective with hash feedback:
let mut objective = feedback_and!(
feedback_or_fast!(CrashFeedback::new(), TimeoutFeedback::new()),
NewHashFeedback::new(&backtrace_observer)
);
This ensures only crashes with unique backtraces are saved.
Dictionary Fuzzing
Use dictionaries to guide fuzzing toward specific tokens:
Add tokens from file:
let mut tokens = Tokens::new();
if let Some(tokenfile) = &tokenfile {
tokens.add_from_file(tokenfile)?;
}
state.add_metadata(tokens);
Update mutator:
let mutator = StdScheduledMutator::new(
havoc_mutations().merge(tokens_mutations())
);
Hard-coded tokens example (PNG):
state.add_metadata(Tokens::from([
vec![137, 80, 78, 71, 13, 10, 26, 10], // PNG header
"IHDR".as_bytes().to_vec(),
"IDAT".as_bytes().to_vec(),
"PLTE".as_bytes().to_vec(),
"IEND".as_bytes().to_vec(),
]));
See Also: For detailed dictionary creation strategies and format-specific dictionaries, see the fuzzing-dictionaries technique skill.
Auto Tokens
Automatically extract magic values and checksums from the program:
Enable in compiler wrapper:
cc.add_pass(LLVMPasses::AutoTokens)
Load auto tokens in fuzzer:
tokens += libafl_targets::autotokens()?;
Verify tokens section:
echo "p (uint8_t *)__token_start" | gdb fuzz
Performance Tuning
| Setting | Impact |
|---|---|
| Multi-core fuzzing | Linear speedup with cores |
InMemoryCorpus | Faster but non-persistent |
InMemoryOnDiskCorpus | Balanced speed and persistence |
| Sanitizers | 2-5x slowdown, essential for bugs |
Optimization level -O2 | Balance between speed and coverage |
Debugging Fuzzer
Run fuzzer in single-process mode for easier debugging:
// Replace launcher with direct call
run_client(None, SimpleEventManager::new(monitor), 0).unwrap();
// Comment out:
// Launcher::builder()
// .run_client(&mut run_client)
// ...
// .launch()
Then debug with GDB:
gdb --args ./fuzz --cores 0 --input corpus/
Real-World Examples
Example: libpng
Fuzzing libpng using LibAFL:
1. Get source code:
curl -L -O https://downloads.sourceforge.net/project/libpng/libpng16/1.6.37/libpng-1.6.37.tar.xz
tar xf libpng-1.6.37.tar.xz
cd libpng-1.6.37/
apt install zlib1g-dev
2. Set compiler wrapper:
export FUZZER_CARGO_DIR="/path/to/libafl/project"
export CC=$FUZZER_CARGO_DIR/target/release/libafl_cc
export CXX=$FUZZER_CARGO_DIR/target/release/libafl_cxx
3. Build static library:
./configure --enable-shared=no
make
4. Get harness:
curl -O https://raw.githubusercontent.com/glennrp/libpng/f8e5fa92b0e37ab597616f554bee254157998227/contrib/oss-fuzz/libpng_read_fuzzer.cc
5. Link fuzzer:
$CXX libpng_read_fuzzer.cc .libs/libpng16.a -lz -o fuzz
6. Prepare seeds:
mkdir seeds/
curl -o seeds/input.png https://raw.githubusercontent.com/glennrp/libpng/acfd50ae0ba3198ad734e5d4dec2b05341e50924/contrib/pngsuite/iftp1n3p08.png
7. Get dictionary (optional):
curl -O https://raw.githubusercontent.com/glennrp/libpng/2fff013a6935967960a5ae626fc21432807933dd/contrib/oss-fuzz/png.dict
8. Start fuzzing:
./fuzz --input seeds/ --cores 0 -x png.dict
Example: CMake Project
Integrate LibAFL with CMake build system:
CMakeLists.txt:
project(BuggyProgram)
cmake_minimum_required(VERSION 3.0)
add_executable(buggy_program main.cc)
add_executable(fuzz main.cc harness.cc)
target_compile_definitions(fuzz PRIVATE NO_MAIN=1)
target_compile_options(fuzz PRIVATE -g -O2)
Build non-instrumented binary:
cmake -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ .
cmake --build . --target buggy_program
Build fuzzer:
export FUZZER_CARGO_DIR="/path/to/libafl/project"
cmake -DCMAKE_C_COMPILER=$FUZZER_CARGO_DIR/target/release/libafl_cc \
-DCMAKE_CXX_COMPILER=$FUZZER_CARGO_DIR/target/release/libafl_cxx .
cmake --build . --target fuzz
Run fuzzing:
./fuzz --input seeds/ --cores 0
Troubleshooting
| Problem | Cause | Solution |
|---|---|---|
| No coverage increases | Instrumentation failed | Verify compiler wrapper used, check for -fsanitize-coverage |
| Fuzzer won't start | Empty corpus with no interesting inputs | Provide seed inputs that trigger code paths |
Linker errors with libafl_main | Runtime not linked | Use -Wl,--whole-archive or -u libafl_main |
| LLVM version mismatch | LibAFL requires LLVM 15-18 | Install compatible LLVM version, set environment variables |
| Rust compilation fails | Outdated Rust or Cargo | Update Rust with rustup update |
| Slow fuzzing | Sanitizers enabled | Expected 2-5x slowdown, necessary for finding bugs |
| Environment variable interference | CC, CXX, RUSTFLAGS set | Unset after building LibAFL project |
| Cannot attach debugger | Multi-process fuzzing | Run in single-process mode (see Debugging section) |
Related Skills
Technique Skills
| Skill | Use Case |
|---|---|
| fuzz-harness-writing | Detailed guidance on writing effective harnesses |
| address-sanitizer | Memory error detection during fuzzing |
| undefined-behavior-sanitizer | Undefined behavior detection |
| coverage-analysis | Measuring and improving code coverage |
| fuzzing-corpus | Building and managing seed corpora |
| fuzzing-dictionaries | Creating dictionaries for format-aware fuzzing |
Related Fuzzers
| Skill | When to Consider |
|---|---|
| libfuzzer | Simpler setup, don't need LibAFL's advanced features |
| aflpp | Multi-core fuzzing without custom fuzzer development |
| cargo-fuzz | Fuzzing Rust projects with less setup |
Resources
Official Documentation
- LibAFL Book - Official handbook with comprehensive documentation
- LibAFL GitHub - Source code and examples
- LibAFL API Documentation - Rust API reference
Examples and Tutorials
- LibAFL Examples - Collection of example fuzzers
- cargo-fuzz with LibAFL - Using LibAFL as cargo-fuzz backend
- Testing Handbook LibAFL Examples - Complete working examples from this handbook
libFuzzer
libFuzzer is an in-process, coverage-guided fuzzer that is part of the LLVM project. It's the recommended starting point for fuzzing C/C++ projects due to its simplicity and integration with the LLVM toolchain. While libFuzzer has been in maintenance-only mode since late 2022, it is easier to install and use than its alternatives, has wide support, and will be maintained for the foreseeable future.
When to Use
| Fuzzer | Best For | Complexity |
|---|---|---|
| libFuzzer | Quick setup, single-project fuzzing | Low |
| AFL++ | Multi-core fuzzing, diverse mutations | Medium |
| LibAFL | Custom fuzzers, research projects | High |
| Honggfuzz | Hardware-based coverage | Medium |
Choose libFuzzer when:
- You need a simple, quick setup for C/C++ code
- Project uses Clang for compilation
- Single-core fuzzing is sufficient initially
- Transitioning to AFL++ later is an option (harnesses are compatible)
Note: Fuzzing harnesses written for libFuzzer are compatible with AFL++, making it easy to transition if you need more advanced features like better multi-core support.
Quick Start
#include <stdint.h>
#include <stddef.h>
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
// Validate input if needed
if (size < 1) return 0;
// Call your target function with fuzzer-provided data
my_target_function(data, size);
return 0;
}
Compile and run:
clang++ -fsanitize=fuzzer,address -g -O2 harness.cc target.cc -o fuzz
mkdir corpus/
./fuzz corpus/
Installation
Prerequisites
- LLVM/Clang compiler (includes libFuzzer)
- LLVM tools for coverage analysis (optional)
Linux (Ubuntu/Debian)
apt install clang llvm
For the latest LLVM version:
# Add LLVM repository from apt.llvm.org
# Then install specific version, e.g.:
apt install clang-18 llvm-18
macOS
# Using Homebrew
brew install llvm
# Or using Nix
nix-env -i clang
Windows
Install Clang through Visual Studio. Refer to Microsoft's documentation for setup instructions.
Recommendation: If possible, fuzz on a local x86_64 VM or rent one on DigitalOcean, AWS, or Hetzner. Linux provides the best support for libFuzzer.
Verification
clang++ --version
# Should show LLVM version information
Writing a Harness
Harness Structure
The harness is the entry point for the fuzzer. libFuzzer calls the LLVMFuzzerTestOneInput function repeatedly with different inputs.
#include <stdint.h>
#include <stddef.h>
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
// 1. Optional: Validate input size
if (size < MIN_REQUIRED_SIZE) {
return 0; // Reject inputs that are too small
}
// 2. Optional: Convert raw bytes to structured data
// Example: Parse two integers from byte array
if (size >= 2 * sizeof(uint32_t)) {
uint32_t a = *(uint32_t*)(data);
uint32_t b = *(uint32_t*)(data + sizeof(uint32_t));
my_function(a, b);
}
// 3. Call target function
target_function(data, size);
// 4. Always return 0 (non-zero reserved for future use)
return 0;
}
Harness Rules
| Do | Don't |
|---|---|
| Handle all input types (empty, huge, malformed) | Call exit() - stops fuzzing process |
| Join all threads before returning | Leave threads running |
| Keep harness fast and simple | Add excessive logging or complexity |
| Maintain determinism | Use random number generators or read /dev/random |
| Reset global state between runs | Rely on state from previous executions |
| Use narrow, focused targets | Mix unrelated data formats (PNG + TCP) in one harness |
Rationale:
- Speed matters: Aim for 100s-1000s executions per second per core
- Reproducibility: Crashes must be reproducible after fuzzing completes
- Isolation: Each execution should be independent
Using FuzzedDataProvider for Complex Inputs
For complex inputs (strings, multiple parameters), use the FuzzedDataProvider helper:
#include <stdint.h>
#include <stddef.h>
#include "FuzzedDataProvider.h" // From LLVM project
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
FuzzedDataProvider fuzzed_data(data, size);
// Extract structured data
size_t allocation_size = fuzzed_data.ConsumeIntegral<size_t>();
std::vector<char> str1 = fuzzed_data.ConsumeBytesWithTerminator<char>(32, 0xFF);
std::vector<char> str2 = fuzzed_data.ConsumeBytesWithTerminator<char>(32, 0xFF);
// Call target with extracted data
char* result = concat(&str1[0], str1.size(), &str2[0], str2.size(), allocation_size);
if (result != NULL) {
free(result);
}
return 0;
}
Download FuzzedDataProvider.h from the LLVM repository.
Interleaved Fuzzing
Use a single harness to test multiple related functions:
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
if (size < 1 + 2 * sizeof(int32_t)) {
return 0;
}
uint8_t mode = data[0];
int32_t numbers[2];
memcpy(numbers, data + 1, 2 * sizeof(int32_t));
// Select function based on first byte
switch (mode % 4) {
case 0: add(numbers[0], numbers[1]); break;
case 1: subtract(numbers[0], numbers[1]); break;
case 2: multiply(numbers[0], numbers[1]); break;
case 3: divide(numbers[0], numbers[1]); break;
}
return 0;
}
See Also: For detailed harness writing techniques, patterns for handling complex inputs, structure-aware fuzzing, and protobuf-based fuzzing, see the fuzz-harness-writing technique skill.
Compilation
Basic Compilation
The key flag is -fsanitize=fuzzer, which:
- Links the libFuzzer runtime (provides
mainfunction) - Enables SanitizerCoverage instrumentation for coverage tracking
- Disables built-in functions like
memcmp
clang++ -fsanitize=fuzzer -g -O2 harness.cc target.cc -o fuzz
Flags explained:
-fsanitize=fuzzer: Enable libFuzzer-g: Add debug symbols (helpful for crash analysis)-O2: Production-level optimizations (recommended for fuzzing)-DNO_MAIN: Define macro if your code has amainfunction
With Sanitizers
AddressSanitizer (recommended):
clang++ -fsanitize=fuzzer,address -g -O2 -U_FORTIFY_SOURCE harness.cc target.cc -o fuzz
Multiple sanitizers:
clang++ -fsanitize=fuzzer,address,undefined -g -O2 harness.cc target.cc -o fuzz
See Also: For detailed sanitizer configuration, common issues, ASAN_OPTIONS flags, and advanced sanitizer usage, see the address-sanitizer and undefined-behavior-sanitizer technique skills.
Build Flags
| Flag | Purpose |
|---|---|
-fsanitize=fuzzer | Enable libFuzzer runtime and instrumentation |
-fsanitize=address | Enable AddressSanitizer (memory error detection) |
-fsanitize=undefined | Enable UndefinedBehaviorSanitizer |
-fsanitize=fuzzer-no-link | Instrument without linking fuzzer (for libraries) |
-g | Include debug symbols |
-O2 | Production optimization level |
-U_FORTIFY_SOURCE | Disable fortification (can interfere with ASan) |
Building Static Libraries
For projects that produce static libraries:
- Build the library with fuzzing instrumentation:
export CC=clang CFLAGS="-fsanitize=fuzzer-no-link -fsanitize=address"
export CXX=clang++ CXXFLAGS="$CFLAGS"
./configure --enable-shared=no
make
- Link the static library with your harness:
clang++ -fsanitize=fuzzer -fsanitize=address harness.cc libmylib.a -o fuzz
CMake Integration
project(FuzzTarget)
cmake_minimum_required(VERSION 3.0)
add_executable(fuzz main.cc harness.cc)
target_compile_definitions(fuzz PRIVATE NO_MAIN=1)
target_compile_options(fuzz PRIVATE -g -O2 -fsanitize=fuzzer -fsanitize=address)
target_link_libraries(fuzz -fsanitize=fuzzer -fsanitize=address)
Build with:
cmake -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ .
cmake --build .
Corpus Management
Creating Initial Corpus
Create a directory for the corpus (can start empty):
mkdir corpus/
Optional but recommended: Provide seed inputs (valid example files):
# For a PNG parser:
cp examples/*.png corpus/
# For a protocol parser:
cp test_packets/*.bin corpus/
Benefits of seed inputs:
- Fuzzer doesn't start from scratch
- Reaches valid code paths faster
- Significantly improves effectiveness
Corpus Structure
The corpus directory contains:
- Input files that trigger unique code paths
- Minimized versions (libFuzzer automatically minimizes)
- Named by content hash (e.g.,
a9993e364706816aba3e25717850c26c9cd0d89d)
Corpus Minimization
libFuzzer automatically minimizes corpus entries during fuzzing. To explicitly minimize:
mkdir minimized_corpus/
./fuzz -merge=1 minimized_corpus/ corpus/
This creates a deduplicated, minimized corpus in minimized_corpus/.
See Also: For corpus creation strategies, seed selection, format-specific corpus building, and corpus maintenance, see the fuzzing-corpus technique skill.
Running Campaigns
Basic Run
./fuzz corpus/
This runs until a crash is found or you stop it (Ctrl+C).
Recommended: Continue After Crashes
./fuzz -fork=1 -ignore_crashes=1 corpus/
The -fork and -ignore_crashes flags (experimental but widely used) allow fuzzing to continue after finding crashes.
Common Options
Control input size:
./fuzz -max_len=4000 corpus/
Rule of thumb: 2x the size of minimal realistic input.
Set timeout:
./fuzz -timeout=2 corpus/
Abort test cases that run longer than 2 seconds.
Use a dictionary:
./fuzz -dict=./format.dict corpus/
Close stdout/stderr (speed up fuzzing):
./fuzz -close_fd_mask=3 corpus/
See all options:
./fuzz -help=1
Multi-Core Fuzzing
Option 1: Jobs and workers (recommended):
./fuzz -jobs=4 -workers=4 -fork=1 -ignore_crashes=1 corpus/
-jobs=4: Run 4 sequential campaigns-workers=4: Process jobs in parallel with 4 processes- Test cases are shared between jobs
Option 2: Fork mode:
./fuzz -fork=4 -ignore_crashes=1 corpus/
Note: For serious multi-core fuzzing, consider switching to AFL++, Honggfuzz, or LibAFL.
Re-executing Test Cases
Re-run a single crash:
./fuzz ./crash-a9993e364706816aba3e25717850c26c9cd0d89d
Test all inputs in a directory without fuzzing:
./fuzz -runs=0 corpus/
Interpreting Output
When fuzzing runs, you'll see statistics like:
INFO: Seed: 3517090860
INFO: Loaded 1 modules (9 inline 8-bit counters)
#2 INITED cov: 3 ft: 4 corp: 1/1b exec/s: 0 rss: 26Mb
#57 NEW cov: 4 ft: 5 corp: 2/4b lim: 4 exec/s: 0 rss: 26Mb
| Output | Meaning |
|---|---|
INITED | Fuzzing initialized |
NEW | New coverage found, added to corpus |
REDUCE | Input minimized while keeping coverage |
cov: N | Number of coverage edges hit |
corp: X/Yb | Corpus size: X entries, Y total bytes |
exec/s: N | Executions per second |
rss: NMb | Resident memory usage |
On crash:
==11672== ERROR: libFuzzer: deadly signal
artifact_prefix='./'; Test unit written to ./crash-a9993e364706816aba3e25717850c26c9cd0d89d
0x61,0x62,0x63,
abc
Base64: YWJj
The crash is saved to ./crash-<hash> with the input shown in hex, UTF-8, and Base64.
Reproducibility: Use -seed=<value> to reproduce a fuzzing campaign (single-core only).
Fuzzing Dictionary
Dictionaries help the fuzzer discover interesting inputs faster by providing hints about the input format.
Dictionary Format
Create a text file with quoted strings (one per line):
# Lines starting with '#' are comments
# Magic bytes
magic="\x89PNG"
magic2="IEND"
# Keywords
"GET"
"POST"
"Content-Type"
# Hex sequences
delimiter="\xFF\xD8\xFF"
Using a Dictionary
./fuzz -dict=./format.dict corpus/
Generating a Dictionary
From header files:
grep -o '".*"' header.h > header.dict
From man pages:
man curl | grep -oP '^\s*(--|-)\K\S+' | sed 's/[,.]$//' | sed 's/^/"&/; s/$/&"/' | sort -u > man.dict
From binary strings:
strings ./binary | sed 's/^/"&/; s/$/&"/' > strings.dict
Using LLMs: Ask ChatGPT or similar to generate a dictionary for your format (e.g., "Generate a libFuzzer dictionary for a JSON parser").
See Also: For advanced dictionary generation, format-specific dictionaries, and dictionary optimization strategies, see the fuzzing-dictionaries technique skill.
Coverage Analysis
While libFuzzer shows basic coverage stats (cov: N), detailed coverage analysis requires additional tools.
Source-Based Coverage
1. Recompile with coverage instrumentation:
clang++ -fsanitize=fuzzer -fprofile-instr-generate -fcoverage-mapping harness.cc target.cc -o fuzz
2. Run fuzzer to collect coverage:
LLVM_PROFILE_FILE="coverage-%p.profraw" ./fuzz -runs=10000 corpus/
3. Merge coverage data:
llvm-profdata merge -sparse coverage-*.profraw -o coverage.profdata
4. Generate coverage report:
llvm-cov show ./fuzz -instr-profile=coverage.profdata
5. Generate HTML report:
llvm-cov show ./fuzz -instr-profile=coverage.profdata -format=html > coverage.html
Improving Coverage
Tips:
- Provide better seed inputs in corpus
- Use dictionaries for format-aware fuzzing
- Check if harness properly exercises target
- Consider structure-aware fuzzing for complex formats
- Run longer campaigns (days/weeks)
See Also: For detailed coverage analysis techniques, identifying coverage gaps, systematic coverage improvement, and comparing coverage across fuzzers, see the coverage-analysis technique skill.
Sanitizer Integration
AddressSanitizer (ASan)
ASan detects memory errors like buffer overflows and use-after-free bugs. Highly recommended for fuzzing.
Enable ASan:
clang++ -fsanitize=fuzzer,address -g -O2 -U_FORTIFY_SOURCE harness.cc target.cc -o fuzz
Example ASan output:
==1276163==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6020000c4ab1
WRITE of size 1 at 0x6020000c4ab1 thread T0
#0 0x55555568631a in check_buf(char*, unsigned long) main.cc:13:25
#1 0x5555556860bf in LLVMFuzzerTestOneInput harness.cc:7:3
Configure ASan with environment variables:
ASAN_OPTIONS=verbosity=1:abort_on_error=1 ./fuzz corpus/
Important flags:
verbosity=1: Show ASan is activedetect_leaks=0: Disable leak detection (leaks reported at end)abort_on_error=1: Callabort()instead of_exit()on errors
Drawbacks:
- 2-4x slowdown
- Requires ~20TB virtual memory (disable memory limits:
-rss_limit_mb=0) - Best supported on Linux
See Also: For comprehensive ASan configuration, common pitfalls, symbolization, and combining with other sanitizers, see the address-sanitizer technique skill.
UndefinedBehaviorSanitizer (UBSan)
UBSan detects undefined behavior like integer overflow, null pointer dereference, etc.
Enable UBSan:
clang++ -fsanitize=fuzzer,undefined -g -O2 harness.cc target.cc -o fuzz
Combine with ASan:
clang++ -fsanitize=fuzzer,address,undefined -g -O2 harness.cc target.cc -o fuzz
MemorySanitizer (MSan)
MSan detects uninitialized memory reads. More complex to use (requires rebuilding all dependencies).
clang++ -fsanitize=fuzzer,memory -g -O2 harness.cc target.cc -o fuzz
Common Sanitizer Issues
| Issue | Solution |
|---|---|
| ASan slows fuzzing too much | Use -fsanitize-recover=address for non-fatal errors |
| Out of memory | Set ASAN_OPTIONS=rss_limit_mb=0 or -rss_limit_mb=0 |
| Stack exhaustion | Increase stack size: ASAN_OPTIONS=stack_size=8388608 |
False positives with _FORTIFY_SOURCE | Use -U_FORTIFY_SOURCE flag |
| MSan reports in dependencies | Rebuild all dependencies with -fsanitize=memory |
Real-World Examples
Example 1: Fuzzing libpng
libpng is a widely-used library for reading/writing PNG images. Bugs can lead to security issues.
1. Get source code:
curl -L -O https://downloads.sourceforge.net/project/libpng/libpng16/1.6.37/libpng-1.6.37.tar.xz
tar xf libpng-1.6.37.tar.xz
cd libpng-1.6.37/
2. Install dependencies:
apt install zlib1g-dev
3. Compile with fuzzing instrumentation:
export CC=clang CFLAGS="-fsanitize=fuzzer-no-link -fsanitize=address"
export CXX=clang++ CXXFLAGS="$CFLAGS"
./configure --enable-shared=no
make
4. Get a harness (or write your own):
curl -O https://raw.githubusercontent.com/glennrp/libpng/f8e5fa92b0e37ab597616f554bee254157998227/contrib/oss-fuzz/libpng_read_fuzzer.cc
5. Prepare corpus and dictionary:
mkdir corpus/
curl -o corpus/input.png https://raw.githubusercontent.com/glennrp/libpng/acfd50ae0ba3198ad734e5d4dec2b05341e50924/contrib/pngsuite/iftp1n3p08.png
curl -O https://raw.githubusercontent.com/glennrp/libpng/2fff013a6935967960a5ae626fc21432807933dd/contrib/oss-fuzz/png.dict
6. Link and compile fuzzer:
clang++ -fsanitize=fuzzer -fsanitize=address libpng_read_fuzzer.cc .libs/libpng16.a -lz -o fuzz
7. Run fuzzing campaign:
./fuzz -close_fd_mask=3 -dict=./png.dict corpus/
Example 2: Simple Division Bug
Harness that finds a division-by-zero bug:
#include <stdint.h>
#include <stddef.h>
double divide(uint32_t numerator, uint32_t denominator) {
// Bug: No check if denominator is zero
return numerator / denominator;
}
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
if(size != 2 * sizeof(uint32_t)) {
return 0;
}
uint32_t numerator = *(uint32_t*)(data);
uint32_t denominator = *(uint32_t*)(data + sizeof(uint32_t));
divide(numerator, denominator);
return 0;
}
Compile and fuzz:
clang++ -fsanitize=fuzzer harness.cc -o fuzz
./fuzz
The fuzzer will quickly find inputs causing a crash.
Advanced Usage
Tips and Tricks
| Tip | Why It Helps |
|---|---|
| Start with single-core, switch to AFL++ for multi-core | libFuzzer harnesses work with AFL++ |
| Use dictionaries for structured formats | 10-100x faster bug discovery |
Close file descriptors with -close_fd_mask=3 | Speed boost if SUT writes output |
Set reasonable -max_len | Prevents wasted time on huge inputs |
| Run for days/weeks, not minutes | Coverage plateaus take time to break |
| Use seed corpus from test suites | Starts fuzzing from valid inputs |
Structure-Aware Fuzzing
For highly structured inputs (e.g., complex protocols, file formats), use libprotobuf-mutator:
- Define input structure using Protocol Buffers
- libFuzzer mutates protobuf messages (structure-preserving mutations)
- Harness converts protobuf to native format
See structure-aware fuzzing documentation for details.
Custom Mutators
libFuzzer allows custom mutators for specialized fuzzing:
extern "C" size_t LLVMFuzzerCustomMutator(uint8_t *Data, size_t Size,
size_t MaxSize, unsigned int Seed) {
// Custom mutation logic
return new_size;
}
extern "C" size_t LLVMFuzzerCustomCrossOver(const uint8_t *Data1, size_t Size1,
const uint8_t *Data2, size_t Size2,
uint8_t *Out, size_t MaxOutSize,
unsigned int Seed) {
// Custom crossover logic
return new_size;
}
Performance Tuning
| Setting | Impact |
|---|---|
-close_fd_mask=3 | Closes stdout/stderr, speeds up fuzzing |
-max_len=<reasonable_size> | Avoids wasting time on huge inputs |
-timeout=<seconds> | Detects hangs, prevents stuck executions |
| Disable ASan for baseline | 2-4x speed boost (but misses memory bugs) |
Use -jobs and -workers | Limited multi-core support |
| Run on Linux | Best platform support and performance |
Troubleshooting
| Problem | Cause | Solution |
|---|---|---|
| No crashes found after hours | Poor corpus, low coverage | Add seed inputs, use dictionary, check harness |
| Very slow executions/sec (<100) | Target too complex, excessive logging | Optimize target, use -close_fd_mask=3, reduce logging |
| Out of memory | ASan's 20TB virtual memory | Set -rss_limit_mb=0 to disable RSS limit |
| Fuzzer stops after first crash | Default behavior | Use -fork=1 -ignore_crashes=1 to continue |
| Can't reproduce crash | Non-determinism in harness/target | Remove random number generation, global state |
Linking errors with -fsanitize=fuzzer | Missing libFuzzer runtime | Ensure using Clang, check LLVM installation |
| GCC project won't compile with Clang | GCC-specific code | Switch to AFL++ with gcc_plugin instead |
| Coverage not improving | Corpus plateau | Run longer, add dictionary, improve seeds, check coverage report |
| Crashes but ASan doesn't trigger | Memory error not detected without ASan | Recompile with -fsanitize=address |
Related Skills
Technique Skills
| Skill | Use Case |
|---|---|
| fuzz-harness-writing | Detailed guidance on writing effective harnesses, structure-aware fuzzing, and FuzzedDataProvider usage |
| address-sanitizer | Memory error detection configuration, ASAN_OPTIONS, and troubleshooting |
| undefined-behavior-sanitizer | Detecting undefined behavior during fuzzing |
| coverage-analysis | Measuring fuzzing effectiveness and identifying untested code paths |
| fuzzing-corpus | Building and managing seed corpora, corpus minimization strategies |
| fuzzing-dictionaries | Creating format-specific dictionaries for faster bug discovery |
Related Fuzzers
| Skill | When to Consider |
|---|---|
| aflpp | When you need serious multi-core fuzzing, or when libFuzzer coverage plateaus |
| honggfuzz | When you want hardware-based coverage feedback on Linux |
| libafl | When building custom fuzzers or conducting fuzzing research |
Resources
Official Documentation
- LLVM libFuzzer Documentation - Official reference
- libFuzzer Tutorial by Google - Step-by-step guide
- SanitizerCoverage - Coverage instrumentation details
Advanced Topics
- Structure-Aware Fuzzing with libprotobuf-mutator
- Split Inputs in libFuzzer
- FuzzedDataProvider Header
Example Projects
- OSS-Fuzz - Continuous fuzzing for open-source projects (many libFuzzer examples)
- AFL++ Dictionary Collection - Reusable dictionaries
OSS-Fuzz
OSS-Fuzz is an open-source project developed by Google that provides free distributed infrastructure for continuous fuzz testing. It streamlines the fuzzing process and facilitates simpler modifications. While only select projects are accepted into OSS-Fuzz, the project's core is open-source, allowing anyone to host their own instance for private projects.
Overview
OSS-Fuzz provides a simple CLI framework for building and starting harnesses or calculating their coverage. Additionally, OSS-Fuzz can be used as a service that hosts static web pages generated from fuzzing outputs such as coverage information.
Key Concepts
| Concept | Description |
|---|---|
| helper.py | CLI script for building images, building fuzzers, and running harnesses locally |
| Base Images | Hierarchical Docker images providing build dependencies and compilers |
| project.yaml | Configuration file defining project metadata for OSS-Fuzz enrollment |
| Dockerfile | Project-specific image with build dependencies |
| build.sh | Script that builds fuzzing harnesses for your project |
| Criticality Score | Metric used by OSS-Fuzz team to evaluate project acceptance |
When to Apply
Apply this technique when:
- Setting up continuous fuzzing for an open-source project
- Need distributed fuzzing infrastructure without managing servers
- Want coverage reports and bug tracking integrated with fuzzing
- Testing existing OSS-Fuzz harnesses locally
- Reproducing crashes from OSS-Fuzz bug reports
Skip this technique when:
- Project is closed-source (unless hosting your own OSS-Fuzz instance)
- Project doesn't meet OSS-Fuzz's criticality score threshold
- Need proprietary or specialized fuzzing infrastructure
- Fuzzing simple scripts that don't warrant infrastructure
Quick Reference
| Task | Command |
|---|---|
| Clone OSS-Fuzz | git clone https://github.com/google/oss-fuzz |
| Build project image | uv run --no-project python infra/helper.py build_image --pull <project> |
| Build fuzzers with ASan | uv run --no-project python infra/helper.py build_fuzzers --sanitizer=address <project> |
| Run specific harness | uv run --no-project python infra/helper.py run_fuzzer <project> <harness> |
| Generate coverage report | uv run --no-project python infra/helper.py coverage <project> |
| Check helper.py options | uv run --no-project python infra/helper.py --help |
OSS-Fuzz Project Components
OSS-Fuzz provides several publicly available tools and web interfaces:
Bug Tracker
The bug tracker allows you to:
- Check bugs from specific projects (initially visible only to maintainers, later made public)
- Create new issues and comment on existing ones
- Search for similar bugs across all projects to understand issues
Build Status System
The build status system helps track:
- Build statuses of all included projects
- Date of last successful build
- Build failures and their duration
Fuzz Introspector
Fuzz Introspector displays:
- Coverage data for projects enrolled in OSS-Fuzz
- Hit frequency for covered code
- Performance analysis and blocker identification
Read this case study for examples and explanations.
Step-by-Step: Running a Single Harness
You don't need to host the whole OSS-Fuzz platform to use it. The helper script makes it easy to run individual harnesses locally.
Step 1: Clone OSS-Fuzz
git clone https://github.com/google/oss-fuzz
cd oss-fuzz
uv run --no-project python infra/helper.py --help
Step 2: Build Project Image
uv run --no-project python infra/helper.py build_image --pull <project-name>
This downloads and builds the base Docker image for the project.
Step 3: Build Fuzzers with Sanitizers
uv run --no-project python infra/helper.py build_fuzzers --sanitizer=address <project-name>
Sanitizer options:
--sanitizer=addressfor AddressSanitizer with LeakSanitizer- Other sanitizers available (language support varies)
Note: Fuzzers are built to /build/out/<project-name>/ containing the harness executables, dictionaries, corpus, and crash files.
Step 4: Run the Fuzzer
uv run --no-project python infra/helper.py run_fuzzer <project-name> <harness-name> [<fuzzer-args>]
The helper script automatically runs any missed steps if you skip them.
Step 5: Coverage Analysis (Optional)
First, install gsutil (skip gcloud initialization).
uv run --no-project python infra/helper.py build_fuzzers --sanitizer=coverage <project-name>
uv run --no-project python infra/helper.py coverage <project-name>
Use --no-corpus-download to use only local corpus. The command generates and hosts a coverage report locally.
See official OSS-Fuzz documentation for details.
Common Patterns
Pattern: Running irssi Example
Use Case: Testing OSS-Fuzz setup with a simple enrolled project
# Clone and navigate to OSS-Fuzz
git clone https://github.com/google/oss-fuzz
cd oss-fuzz
# Build and run irssi fuzzer
uv run --no-project python infra/helper.py build_image --pull irssi
uv run --no-project python infra/helper.py build_fuzzers --sanitizer=address irssi
uv run --no-project python infra/helper.py run_fuzzer irssi irssi-fuzz
Expected Output:
INFO:__main__:Running: docker run --rm --privileged --shm-size=2g --platform linux/amd64 -i -e FUZZING_ENGINE=libfuzzer -e SANITIZER=address -e RUN_FUZZER_MODE=interactive -e HELPER=True -v /private/tmp/oss-fuzz/build/out/irssi:/out -t gcr.io/oss-fuzz-base/base-runner run_fuzzer irssi-fuzz.
Using seed corpus: irssi-fuzz_seed_corpus.zip
/out/irssi-fuzz -rss_limit_mb=2560 -timeout=25 /tmp/irssi-fuzz_corpus -max_len=2048 < /dev/null
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1531341664
INFO: Loaded 1 modules (95687 inline 8-bit counters): 95687 [0x1096c80, 0x10ae247),
INFO: Loaded 1 PC tables (95687 PCs): 95687 [0x10ae248,0x1223eb8),
INFO: 719 files found in /tmp/irssi-fuzz_corpus
INFO: seed corpus: files: 719 min: 1b max: 170106b total: 367969b rss: 48Mb
#720 INITED cov: 409 ft: 1738 corp: 640/163Kb exec/s: 0 rss: 62Mb
#762 REDUCE cov: 409 ft: 1738 corp: 640/163Kb lim: 2048 exec/s: 0 rss: 63Mb L: 236/2048 MS: 2 ShuffleBytes-EraseBytes-
Pattern: Enrolling a New Project
Use Case: Adding your project to OSS-Fuzz (or private instance)
Create three files in projects/<your-project>/:
1. project.yaml - Project metadata:
homepage: "https://github.com/yourorg/yourproject"
language: c++
primary_contact: "your-email@example.com"
main_repo: "https://github.com/yourorg/yourproject"
fuzzing_engines:
- libfuzzer
sanitizers:
- address
- undefined
2. Dockerfile - Build dependencies:
FROM gcr.io/oss-fuzz-base/base-builder
RUN apt-get update && apt-get install -y \
autoconf \
automake \
libtool \
pkg-config
RUN git clone --depth 1 https://github.com/yourorg/yourproject
WORKDIR yourproject
COPY build.sh $SRC/
3. build.sh - Build harnesses:
#!/bin/bash -eu
./autogen.sh
./configure --disable-shared
make -j$(nproc)
# Build harnesses
$CXX $CXXFLAGS -std=c++11 -I. \
$SRC/yourproject/fuzz/harness.cc -o $OUT/harness \
$LIB_FUZZING_ENGINE ./libyourproject.a
# Copy corpus and dictionary if available
cp $SRC/yourproject/fuzz/corpus.zip $OUT/harness_seed_corpus.zip
cp $SRC/yourproject/fuzz/dictionary.dict $OUT/harness.dict
Docker Images in OSS-Fuzz
Harnesses are built and executed in Docker containers. All projects share a runner image, but each project has its own build image.
Image Hierarchy
Images build on each other in this sequence:
- base_image - Specific Ubuntu version
- base_clang - Clang compiler; based on
base_image - base_builder - Build dependencies; based on
base_clang- Language-specific variants:
base_builder_go, etc. - See /oss-fuzz/infra/base-images/ for full list
- Language-specific variants:
- Your project Docker image - Project-specific dependencies; based on
base_builderor language variant
Runner Images (Used Separately)
- base_runner - Executes harnesses; based on
base_clang - base_runner_debug - With debug tools; based on
base_runner
Advanced Usage
Tips and Tricks
| Tip | Why It Helps |
|---|---|
| Don't manually copy source code | Project Dockerfile likely already pulls latest version |
| Check existing projects | Browse oss-fuzz/projects for examples |
| Keep harnesses in separate repo | Like curl-fuzzer - cleaner organization |
| Use specific compiler versions | Base images provide consistent build environment |
| Install dependencies in Dockerfile | May require approval for OSS-Fuzz enrollment |
Criticality Score
OSS-Fuzz uses a criticality score to evaluate project acceptance. See this example for how scoring works.
Projects with lower scores may still be added to private OSS-Fuzz instances.
Hosting Your Own Instance
Since OSS-Fuzz is open-source, you can host your own instance for:
- Private projects not eligible for public OSS-Fuzz
- Projects with lower criticality scores
- Custom fuzzing infrastructure needs
Anti-Patterns
| Anti-Pattern | Problem | Correct Approach |
|---|---|---|
| Manually pulling source in build.sh | Doesn't use latest version | Let Dockerfile handle git clone |
| Copying code to OSS-Fuzz repo | Hard to maintain, violates separation | Reference external harness repo |
| Ignoring base image versions | Build inconsistencies | Use provided base images and compilers |
| Skipping local testing | Wastes CI resources | Use helper.py locally before PR |
| Not checking build status | Unnoticed build failures | Monitor build status page regularly |
Tool-Specific Guidance
libFuzzer
OSS-Fuzz primarily uses libFuzzer as the fuzzing engine for C/C++ projects.
Harness signature:
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
// Your fuzzing logic
return 0;
}
Build in build.sh:
$CXX $CXXFLAGS -std=c++11 -I. \
harness.cc -o $OUT/harness \
$LIB_FUZZING_ENGINE ./libproject.a
Integration tips:
- Use
$LIB_FUZZING_ENGINEvariable provided by OSS-Fuzz - Include
-fsanitize=fuzzeris handled automatically - Link against static libraries when possible
AFL++
OSS-Fuzz supports AFL++ as an alternative fuzzing engine.
Enable in project.yaml:
fuzzing_engines:
- afl
- libfuzzer
Integration tips:
- AFL++ harnesses work alongside libFuzzer harnesses
- Use persistent mode for better performance
- OSS-Fuzz handles engine-specific compilation flags
Atheris (Python)
For Python projects with C extensions.
Example from cbor2 integration:
Harness:
import atheris
import sys
import cbor2
@atheris.instrument_func
def TestOneInput(data):
fdp = atheris.FuzzedDataProvider(data)
try:
cbor2.loads(data)
except (cbor2.CBORDecodeError, ValueError):
pass
def main():
atheris.Setup(sys.argv, TestOneInput)
atheris.Fuzz()
if __name__ == "__main__":
main()
Build in build.sh:
# allow-legacy-python: build.sh runs inside the oss-fuzz container, where the shims are absent.
pip3 install .
for fuzzer in $(find $SRC -name 'fuzz_*.py'); do
compile_python_fuzzer $fuzzer
done
Integration tips:
- Use
compile_python_fuzzerhelper provided by OSS-Fuzz - See Continuously Fuzzing Python C Extensions blog post
Rust Projects
Enable in project.yaml:
language: rust
fuzzing_engines:
- libfuzzer
sanitizers:
- address # Only AddressSanitizer supported for Rust
Build in build.sh:
cargo fuzz build -O --debug-assertions
cp fuzz/target/x86_64-unknown-linux-gnu/release/fuzz_target_1 $OUT/
Integration tips:
- Rust supports only AddressSanitizer with libfuzzer
- Use cargo-fuzz for local development
- OSS-Fuzz handles Rust-specific compilation
Troubleshooting
| Issue | Cause | Solution |
|---|---|---|
| Build fails with missing dependencies | Dependencies not in Dockerfile | Add apt-get install or equivalent in Dockerfile |
| Harness crashes immediately | Missing input validation | Add size checks in harness |
| Coverage is 0% | Harness not reaching target code | Verify harness actually calls target functions |
| Build timeout | Complex build process | Optimize build.sh, consider parallel builds |
| Sanitizer errors in build | Incompatible flags | Use flags provided by OSS-Fuzz environment variables |
| Cannot find source code | Wrong working directory in Dockerfile | Set WORKDIR or use absolute paths |
Related Skills
Tools That Use This Technique
| Skill | How It Applies |
|---|---|
| libfuzzer | Primary fuzzing engine used by OSS-Fuzz |
| aflpp | Alternative fuzzing engine supported by OSS-Fuzz |
| atheris | Used for fuzzing Python projects in OSS-Fuzz |
| cargo-fuzz | Used for Rust projects in OSS-Fuzz |
Related Techniques
| Skill | Relationship |
|---|---|
| coverage-analysis | OSS-Fuzz generates coverage reports via helper.py |
| address-sanitizer | Default sanitizer for OSS-Fuzz projects |
| fuzz-harness-writing | Essential for enrolling projects in OSS-Fuzz |
| corpus-management | OSS-Fuzz maintains corpus for enrolled projects |
Resources
Key External Resources
OSS-Fuzz Official Documentation Comprehensive documentation covering enrollment, harness writing, and troubleshooting for the OSS-Fuzz platform.
Getting Started Guide Step-by-step process for enrolling new projects into OSS-Fuzz, including requirements and approval process.
cbor2 OSS-Fuzz Integration PR Real-world example of enrolling a Python project with C extensions into OSS-Fuzz. Shows:
- Initial proposal and project introduction
- Criticality score evaluation
- Complete implementation (project.yaml, Dockerfile, build.sh, harnesses)
Fuzz Introspector Case Studies Examples and explanations of using Fuzz Introspector to analyze coverage and identify fuzzing blockers.
Video Resources
Check OSS-Fuzz documentation for workshop recordings and tutorials on enrollment and harness development.
Property-Based Testing
An example test asserts one point. A property asserts a rule over the whole input domain and lets the generator hunt for the counterexample. That trade is worth making when the code has an algebraic shape — an inverse, an invariant, an oracle — and not otherwise. Code with no such shape gets example tests; saying so is a valid outcome.
Check first whether the shape is missing or merely buried. A calculation wrapped in I/O, a string built by concatenation, an in-place mutation — each has a property and no seam to assert it through. See references/refactoring.md before concluding there is nothing to assert.
Property catalog
| Property | Formula | Where it applies |
|---|---|---|
| Roundtrip | decode(encode(x)) == x | Serialization, conversion pairs |
| Inverse | f(g(x)) == x | encrypt/decrypt, compress/decompress |
| Oracle | new(x) == reference(x) | Optimization, refactoring, reimplementation |
| Idempotence | f(f(x)) == f(x) | Normalization, formatting, sorting |
| Invariant | Holds before and after | Any transformation, contract state |
| Easy to verify | is_sorted(sort(x)) | Complex algorithms with cheap checkers |
| Commutativity | f(a, b) == f(b, a) | Binary and set operations |
| Associativity | f(f(a,b), c) == f(a, f(b,c)) | Combining operations |
| Identity | f(x, e) == x | Operations with a neutral element |
Strength ordering, weakest to strongest:
no crash → type preservation → invariant → idempotence → roundtrip / oracle.
Assert the strongest property the code supports. "No crash" alone rarely justifies the dependency — if that is all you can find, either a small rearrangement exposes something stronger, or the honest report is that this code is a poor PBT candidate. Rule out the first before settling for the second.
The two ways a property test asserts nothing
- Tautology.
assert add(a, b) == a + brestates the implementation; no bug they share can fail it. Pick a property that constrains the function without recomputing it. Note the exception:f(x) == f(x)is a genuine determinism property whenfis not obviously pure — serializers over dicts or sets, hashing, anything reading the clock. - Vacuity.
assume()that filters out nearly every input passes without exercising anything, and self-contradictoryassume()passes having run zero cases. Push constraints into the strategy so the generator produces valid inputs directly.
Where to look next
Load the one that matches the task in front of you:
| Task | File |
|---|---|
| Writing new tests, designing strategies | references/generating.md |
| The code has no property to assert yet | references/refactoring.md |
| Reviewing existing property tests | references/reviewing.md |
| A property test just failed | references/interpreting-failures.md |
| Library choice, Echidna and Medusa | references/libraries.md |
Introducing PBT to a project that lacks it
If the project already uses a PBT library, just write the tests in it. If it does not, adding one is a dependency decision that belongs to the user — offer it once with the specific property you would write, and take the answer either way.
Ruzzy
Ruzzy is a coverage-guided fuzzer for Ruby built on libFuzzer. It enables fuzzing both pure Ruby code and Ruby C extensions with sanitizer support for detecting memory corruption and undefined behavior.
When to Use
Ruzzy is currently the only production-ready coverage-guided fuzzer for Ruby.
Choose Ruzzy when:
- Fuzzing Ruby applications or libraries
- Testing Ruby C extensions for memory safety issues
- You need coverage-guided fuzzing for Ruby code
- Working with Ruby gems that have native extensions
Quick Start
Set up environment:
export ASAN_OPTIONS="allocator_may_return_null=1:detect_leaks=0:use_sigaltstack=0"
Test with the included toy example:
LD_PRELOAD=$(ruby -e 'require "ruzzy"; print Ruzzy::ASAN_PATH') \
ruby -e 'require "ruzzy"; Ruzzy.dummy'
This should quickly find a crash demonstrating that Ruzzy is working correctly.
Installation
Platform Support
Ruzzy supports Linux x86-64 and AArch64/ARM64. For macOS or Windows, use the Dockerfile or development environment.
Prerequisites
- Linux x86-64 or AArch64/ARM64
- Recent version of clang (tested back to 14.0.0, latest release recommended)
- Ruby with gem installed
Installation Command
Install Ruzzy with clang compiler flags:
MAKE="make --environment-overrides V=1" \
CC="/path/to/clang" \
CXX="/path/to/clang++" \
LDSHARED="/path/to/clang -shared" \
LDSHAREDXX="/path/to/clang++ -shared" \
gem install ruzzy
Environment variables explained:
MAKE: Overrides make to respect subsequent environment variablesCC,CXX,LDSHARED,LDSHAREDXX: Ensure proper clang binaries are used for latest features
Troubleshooting Installation
If installation fails, enable debug output:
RUZZY_DEBUG=1 gem install --verbose ruzzy
Verification
Verify installation by running the toy example (see Quick Start section).
Writing a Harness
Fuzzing Pure Ruby Code
Pure Ruby fuzzing requires two scripts due to Ruby interpreter implementation details.
Tracer script (test_tracer.rb):
# frozen_string_literal: true
require 'ruzzy'
Ruzzy.trace('test_harness.rb')
Harness script (test_harness.rb):
# frozen_string_literal: true
require 'ruzzy'
def fuzzing_target(input)
# Your code to fuzz here
if input.length == 4
if input[0] == 'F'
if input[1] == 'U'
if input[2] == 'Z'
if input[3] == 'Z'
raise
end
end
end
end
end
end
test_one_input = lambda do |data|
fuzzing_target(data)
return 0
end
Ruzzy.fuzz(test_one_input)
Run with:
LD_PRELOAD=$(ruby -e 'require "ruzzy"; print Ruzzy::ASAN_PATH') \
ruby test_tracer.rb
Fuzzing Ruby C Extensions
C extensions can be fuzzed with a single harness file, no tracer needed.
Example harness for msgpack (fuzz_msgpack.rb):
# frozen_string_literal: true
require 'msgpack'
require 'ruzzy'
test_one_input = lambda do |data|
begin
MessagePack.unpack(data)
rescue Exception
# We're looking for memory corruption, not Ruby exceptions
end
return 0
end
Ruzzy.fuzz(test_one_input)
Run with:
LD_PRELOAD=$(ruby -e 'require "ruzzy"; print Ruzzy::ASAN_PATH') \
ruby fuzz_msgpack.rb
Harness Rules
| Do | Don't |
|---|---|
| Catch Ruby exceptions if testing C extensions | Let Ruby exceptions crash the fuzzer |
| Return 0 from test_one_input lambda | Return other values |
| Keep harness deterministic | Use randomness or time-based logic |
| Use tracer script for pure Ruby | Skip tracer for pure Ruby code |
See Also: For detailed harness writing techniques, patterns for handling complex inputs, and advanced strategies, see the fuzz-harness-writing technique skill.
Compilation
Installing Gems with Sanitizers
When installing Ruby gems with C extensions for fuzzing, compile with sanitizer flags:
MAKE="make --environment-overrides V=1" \
CC="/path/to/clang" \
CXX="/path/to/clang++" \
LDSHARED="/path/to/clang -shared" \
LDSHAREDXX="/path/to/clang++ -shared" \
CFLAGS="-fsanitize=address,fuzzer-no-link -fno-omit-frame-pointer -fno-common -fPIC -g" \
CXXFLAGS="-fsanitize=address,fuzzer-no-link -fno-omit-frame-pointer -fno-common -fPIC -g" \
gem install <gem-name>
Build Flags
| Flag | Purpose |
|---|---|
-fsanitize=address,fuzzer-no-link | Enable AddressSanitizer and fuzzer instrumentation |
-fno-omit-frame-pointer | Improve stack trace quality |
-fno-common | Better compatibility with sanitizers |
-fPIC | Position-independent code for shared libraries |
-g | Include debug symbols |
Running Campaigns
Environment Setup
Before running any fuzzing campaign, set ASAN_OPTIONS:
export ASAN_OPTIONS="allocator_may_return_null=1:detect_leaks=0:use_sigaltstack=0"
Options explained:
allocator_may_return_null=1: Skip common low-impact allocation failures (DoS)detect_leaks=0: Ruby interpreter leaks data, ignore these for nowuse_sigaltstack=0: Ruby recommends disabling sigaltstack with ASan
Basic Run
LD_PRELOAD=$(ruby -e 'require "ruzzy"; print Ruzzy::ASAN_PATH') \
ruby harness.rb
Note: LD_PRELOAD is required for sanitizer injection. Unlike ASAN_OPTIONS, do not export it as it may interfere with other programs.
With Corpus
LD_PRELOAD=$(ruby -e 'require "ruzzy"; print Ruzzy::ASAN_PATH') \
ruby harness.rb /path/to/corpus
Passing libFuzzer Options
All libFuzzer options can be passed as arguments:
LD_PRELOAD=$(ruby -e 'require "ruzzy"; print Ruzzy::ASAN_PATH') \
ruby harness.rb /path/to/corpus -max_len=1024 -timeout=10
See libFuzzer options for full reference.
Reproducing Crashes
Re-run a crash case by passing the crash file:
LD_PRELOAD=$(ruby -e 'require "ruzzy"; print Ruzzy::ASAN_PATH') \
ruby harness.rb ./crash-253420c1158bc6382093d409ce2e9cff5806e980
Interpreting Output
| Output | Meaning |
|---|---|
INFO: Running with entropic power schedule | Fuzzing campaign started |
ERROR: AddressSanitizer: heap-use-after-free | Memory corruption detected |
SUMMARY: libFuzzer: fuzz target exited | Ruby exception occurred |
artifact_prefix='./'; Test unit written to ./crash-* | Crash input saved |
Base64: ... | Base64 encoding of crash input |
Sanitizer Integration
AddressSanitizer (ASan)
Ruzzy includes a pre-compiled AddressSanitizer library:
LD_PRELOAD=$(ruby -e 'require "ruzzy"; print Ruzzy::ASAN_PATH') \
ruby harness.rb
Use ASan for detecting:
- Heap buffer overflows
- Stack buffer overflows
- Use-after-free
- Double-free
- Memory leaks (disabled by default in Ruzzy)
UndefinedBehaviorSanitizer (UBSan)
Ruzzy also includes UBSan:
LD_PRELOAD=$(ruby -e 'require "ruzzy"; print Ruzzy::UBSAN_PATH') \
ruby harness.rb
Use UBSan for detecting:
- Signed integer overflow
- Null pointer dereferences
- Misaligned memory access
- Division by zero
Common Sanitizer Issues
| Issue | Solution |
|---|---|
| Ruby interpreter leak warnings | Use ASAN_OPTIONS=detect_leaks=0 |
| Sigaltstack conflicts | Use ASAN_OPTIONS=use_sigaltstack=0 |
| Allocation failure spam | Use ASAN_OPTIONS=allocator_may_return_null=1 |
| LD_PRELOAD interferes with tools | Don't export it; set inline with ruby command |
See Also: For detailed sanitizer configuration, common issues, and advanced flags, see the address-sanitizer and undefined-behavior-sanitizer technique skills.
Real-World Examples
Example: msgpack-ruby
Fuzzing the msgpack MessagePack parser for memory corruption.
Install with sanitizers:
MAKE="make --environment-overrides V=1" \
CC="/path/to/clang" \
CXX="/path/to/clang++" \
LDSHARED="/path/to/clang -shared" \
LDSHAREDXX="/path/to/clang++ -shared" \
CFLAGS="-fsanitize=address,fuzzer-no-link -fno-omit-frame-pointer -fno-common -fPIC -g" \
CXXFLAGS="-fsanitize=address,fuzzer-no-link -fno-omit-frame-pointer -fno-common -fPIC -g" \
gem install msgpack
Harness (fuzz_msgpack.rb):
# frozen_string_literal: true
require 'msgpack'
require 'ruzzy'
test_one_input = lambda do |data|
begin
MessagePack.unpack(data)
rescue Exception
# We're looking for memory corruption, not Ruby exceptions
end
return 0
end
Ruzzy.fuzz(test_one_input)
Run:
export ASAN_OPTIONS="allocator_may_return_null=1:detect_leaks=0:use_sigaltstack=0"
LD_PRELOAD=$(ruby -e 'require "ruzzy"; print Ruzzy::ASAN_PATH') \
ruby fuzz_msgpack.rb
Example: Pure Ruby Target
Fuzzing pure Ruby code with a custom parser.
Tracer (test_tracer.rb):
# frozen_string_literal: true
require 'ruzzy'
Ruzzy.trace('test_harness.rb')
Harness (test_harness.rb):
# frozen_string_literal: true
require 'ruzzy'
require_relative 'my_parser'
test_one_input = lambda do |data|
begin
MyParser.parse(data)
rescue StandardError
# Expected exceptions from malformed input
end
return 0
end
Ruzzy.fuzz(test_one_input)
Run:
export ASAN_OPTIONS="allocator_may_return_null=1:detect_leaks=0:use_sigaltstack=0"
LD_PRELOAD=$(ruby -e 'require "ruzzy"; print Ruzzy::ASAN_PATH') \
ruby test_tracer.rb
Troubleshooting
| Problem | Cause | Solution |
|---|---|---|
| Installation fails | Wrong clang version or path | Verify clang path, use clang 14.0.0+ |
cannot open shared object file | LD_PRELOAD not set | Set LD_PRELOAD inline with ruby command |
| Fuzzer immediately exits | Missing corpus directory | Create corpus directory or pass as argument |
| No coverage progress | Pure Ruby needs tracer | Use tracer script for pure Ruby code |
| Leak detection spam | Ruby interpreter leaks | Set ASAN_OPTIONS=detect_leaks=0 |
| Installation debug needed | Compilation errors | Use RUZZY_DEBUG=1 gem install --verbose ruzzy |
Related Skills
Technique Skills
| Skill | Use Case |
|---|---|
| fuzz-harness-writing | Detailed guidance on writing effective harnesses |
| address-sanitizer | Memory error detection during fuzzing |
| undefined-behavior-sanitizer | Detecting undefined behavior in C extensions |
| libfuzzer | Understanding libFuzzer options (Ruzzy is built on libFuzzer) |
Related Fuzzers
| Skill | When to Consider |
|---|---|
| libfuzzer | When fuzzing Ruby C extension code directly in C/C++ |
| aflpp | Alternative approach for fuzzing Ruby by instrumenting Ruby interpreter |
Resources
Key External Resources
Introducing Ruzzy, a coverage-guided Ruby fuzzer Official Trail of Bits blog post announcing Ruzzy, covering motivation, architecture, and initial results.
Ruzzy GitHub Repository Source code, additional examples, and development instructions.
libFuzzer Documentation Since Ruzzy is built on libFuzzer, understanding libFuzzer options and behavior is valuable.
Fuzzing Ruby C extensions Detailed guide on fuzzing C extensions with compilation flags and examples.
Fuzzing pure Ruby code Detailed guide on the tracer pattern required for pure Ruby fuzzing.
Testing Handbook Skill Generator
Generate and maintain Claude Code skills from the Trail of Bits Testing Handbook.
When to Use
Invoke this skill when:
- Creating new security testing skills from handbook content
- User mentions "testing handbook", "appsec.guide", or asks about generating skills
- Bulk skill generation or refresh is needed
Do NOT use for:
- General security testing questions (use the generated skills)
- Non-handbook skill creation
Handbook Location
The skill needs the Testing Handbook repository. See discovery.md for full details.
Quick reference: Check ./testing-handbook, ../testing-handbook, ~/testing-handbook → ask user → clone as last resort.
Repository: https://github.com/trailofbits/testing-handbook
Workflow Overview
Phase 0: Setup Phase 1: Discovery
┌─────────────────┐ ┌─────────────────┐
│ Locate handbook │ → │ Analyze handbook│
│ - Find or clone │ │ - Scan sections │
│ - Confirm path │ │ - Classify types│
└─────────────────┘ └─────────────────┘
↓ ↓
Phase 3: Generation Phase 2: Planning
┌─────────────────┐ ┌─────────────────┐
│ TWO-PASS GEN │ ← │ Generate plan │
│ Pass 1: Content │ │ - New skills │
│ Pass 2: X-refs │ │ - Updates │
│ - Write to gen/ │ │ - Present user │
└─────────────────┘ └─────────────────┘
↓
Phase 4: Testing Phase 5: Finalize
┌─────────────────┐ ┌─────────────────┐
│ Validate skills │ → │ Post-generation │
│ - Run validator │ │ - Update README │
│ - Test activation│ │ - Update X-refs │
│ - Fix issues │ │ - Self-improve │
└─────────────────┘ └─────────────────┘
Scope Restrictions
ONLY modify these locations:
plugins/testing-handbook-skills/skills/[skill-name]/*- Generated skills (as siblings to testing-handbook-generator)plugins/testing-handbook-skills/skills/testing-handbook-generator/*- Self-improvement- Repository root
README.md- Add generated skills to table
NEVER modify or analyze:
- Other plugins (
plugins/property-based-testing/,plugins/static-analysis/, etc.) - Other skills outside this plugin
Do not scan or pull into context any skills outside of testing-handbook-skills/. Generate skills based solely on handbook content and resources referenced from it.
Quick Reference
Section → Skill Type Mapping
| Handbook Section | Skill Type | Template |
|---|---|---|
/static-analysis/[tool]/ | Tool Skill | tool-skill.md |
/fuzzing/[lang]/[fuzzer]/ | Fuzzer Skill | fuzzer-skill.md |
/fuzzing/techniques/ | Technique Skill | technique-skill.md |
/crypto/[tool]/ | Domain Skill | domain-skill.md |
/web/[tool]/ | Tool Skill | tool-skill.md |
Skill Candidate Signals
| Signal | Indicates |
|---|---|
_index.md with bookCollapseSection: true | Major tool/topic |
| Numbered files (00-, 10-, 20-) | Structured content |
techniques/ subsection | Methodology content |
99-resources.md or 91-resources.md | Has external links |
Exclusion Signals
| Signal | Action |
|---|---|
draft: true in frontmatter | Skip section |
| Empty directory | Skip section |
| Template/placeholder file | Skip section |
GUI-only tool (e.g., web/burp/) | Skip section (Claude cannot operate GUI tools) |
Decision Tree
Starting skill generation?
├─ Need to analyze handbook and build plan?
│ └─ Read: discovery.md
│ (Handbook analysis methodology, plan format)
│
├─ Spawning skill generation agents?
│ └─ Read: agent-prompt.md
│ (Full prompt template, variable reference, validation checklist)
│
├─ Generating a specific skill type?
│ └─ Read appropriate template:
│ ├─ Tool (Semgrep, CodeQL) → templates/tool-skill.md
│ ├─ Fuzzer (libFuzzer, AFL++) → templates/fuzzer-skill.md
│ ├─ Technique (harness, coverage) → templates/technique-skill.md
│ └─ Domain (crypto, web) → templates/domain-skill.md
│
├─ Validating generated skills?
│ └─ Run: scripts/validate-skills.py
│ Then read: testing.md for activation testing
│
├─ Finalizing after generation?
│ └─ See: Post-Generation Tasks below
│ (Update main README, update Skills Cross-Reference, self-improvement)
│
└─ Quick generation from specific section?
└─ Use Quick Reference above, apply template directly
Two-Pass Generation (Phase 3)
Generation uses a two-pass approach to solve forward reference problems (skills referencing other skills that don't exist yet).
Pass 1: Content Generation (Parallel)
Generate all skills in parallel without the Related Skills section:
Pass 1 - Generating 5 skills in parallel:
├─ Agent 1: libfuzzer (fuzzer) → skills/libfuzzer/SKILL.md
├─ Agent 2: aflpp (fuzzer) → skills/aflpp/SKILL.md
├─ Agent 3: semgrep (tool) → skills/semgrep/SKILL.md
├─ Agent 4: harness-writing (technique) → skills/harness-writing/SKILL.md
└─ Agent 5: wycheproof (domain) → skills/wycheproof/SKILL.md
Each agent uses: pass=1 (content only, Related Skills left empty)
Pass 1 agents:
- Generate all sections EXCEPT Related Skills
- Leave a placeholder:
## Related Skills\n\n<!-- PASS2: populate after all skills exist --> - Output report includes
references: DEFERRED
Pass 2: Cross-Reference Population (Sequential)
After all Pass 1 agents complete, run Pass 2 to populate Related Skills:
Pass 2 - Populating cross-references:
├─ Read all generated skill names from skills/*/SKILL.md
├─ For each skill, determine related skills based on:
│ ├─ related_sections from discovery (handbook structure)
│ ├─ Skill type relationships (fuzzers → techniques)
│ └─ Explicit mentions in content
└─ Update each SKILL.md's Related Skills section
Pass 2 process:
- Collect all generated skill names:
ls -d skills/*/SKILL.md - For each skill, identify related skills using the mapping from discovery
- Edit each SKILL.md to replace the placeholder with actual links
- Validate cross-references exist (no broken links)
Agent Prompt Template
See agent-prompt.md for the full prompt template with:
- Variable substitution reference (including
passvariable) - Pre-write validation checklist
- Hugo shortcode conversion rules
- Line count splitting rules
- Error handling guidance
- Output report format
Collecting Results
After Pass 1: Aggregate output reports, verify all skills generated. After Pass 2: Run validator to check cross-references.
Handling Agent Failures
If an agent fails or produces invalid output:
| Failure Type | Detection | Recovery Action |
|---|---|---|
| Agent crashed | No output report | Re-run single agent with same inputs |
| Validation failed | Output report shows errors | Check gaps/warnings, manually patch or re-run |
| Wrong skill type | Content doesn't match template | Re-run with corrected type parameter |
| Missing content | Output report lists gaps | Accept if minor, or provide additional related_sections |
| Pass 2 broken ref | Validator shows missing skill | Check if skill was skipped, update reference |
Important: Do NOT re-run the entire parallel batch for a single agent failure. Fix individual failures independently.
Single-Skill Regeneration
To regenerate a single skill without re-running the entire batch:
# Regenerate single skill (Pass 1 - content only)
"Use testing-handbook-generator to regenerate the {skill-name} skill from section {section_path}"
# Example:
"Use testing-handbook-generator to regenerate the libfuzzer skill from section fuzzing/c-cpp/10-libfuzzer"
Regeneration workflow:
- Re-read the handbook section for fresh content
- Apply the appropriate template
- Write to
skills/{skill-name}/SKILL.md(overwrites existing) - Re-run Pass 2 for that skill only to update cross-references
- Run validator on the single skill:
uv run scripts/validate-skills.py --skill {skill-name}
Output Location
Generated skills are written to:
skills/[skill-name]/SKILL.md
Each skill gets its own directory for potential supporting files (as siblings to testing-handbook-generator).
Quality Checklist
Before delivering generated skills:
- All handbook sections analyzed (Phase 1)
- Plan presented to user before generation (Phase 2)
- Parallel agents launched - one per skill (Phase 3)
- Templates applied correctly per skill type
- Validator passes:
uv run scripts/validate-skills.py - Activation testing passed - see testing.md
- Main
README.mdupdated with generated skills table -
README.mdSkills Cross-Reference graph updated - Self-improvement notes captured
- User notified with summary
Post-Generation Tasks
1. Update Main README
After generating skills, update the repository's main README.md to list them.
Format: Add generated skills to the same "Available Plugins" table, directly after testing-handbook-skills. Use plain text testing-handbook-generator as the author (no link).
Example:
| Plugin | Description | Author |
|--------|-------------|--------|
| ... other plugins ... |
| [testing-handbook-skills](plugins/testing-handbook-skills/) | Meta-skill that generates skills from the Testing Handbook | Paweł Płatek |
| [libfuzzer](plugins/testing-handbook-skills/skills/libfuzzer/) | Coverage-guided fuzzing with libFuzzer for C/C++ | testing-handbook-generator |
| [aflpp](plugins/testing-handbook-skills/skills/aflpp/) | Multi-core fuzzing with AFL++ | testing-handbook-generator |
| [semgrep](plugins/testing-handbook-skills/skills/semgrep/) | Fast static analysis for finding bugs | testing-handbook-generator |
2. Update Skills Cross-Reference
After generating skills, update the README.md's Skills Cross-Reference section with the mermaid graph showing skill relationships.
Process:
- Read each generated skill's
SKILL.mdand extract its## Related Skillssection - Build the mermaid graph with nodes grouped by skill type (Fuzzers, Techniques, Tools, Domain)
- Add edges based on the Related Skills relationships:
- Solid arrows (
-->) for primary technique dependencies - Dashed arrows (
-.->) for alternative tool suggestions
- Solid arrows (
- Replace the existing mermaid code block in README.md
Edge classification:
| Relationship | Arrow Style | Example |
|---|---|---|
| Fuzzer → Technique | --> | libfuzzer --> harness-writing |
| Tool → Tool (alternative) | -.-> | semgrep -.-> codeql |
| Fuzzer → Fuzzer (alternative) | -.-> | libfuzzer -.-> aflpp |
| Technique → Technique | --> | harness-writing --> coverage-analysis |
Validation: After updating, run validate-skills.py to verify all referenced skills exist.
3. Self-Improvement
After each generation run, reflect on what could improve future runs.
Capture improvements to:
- Templates (missing sections, better structure)
- Discovery logic (missed patterns, false positives)
- Content extraction (shortcodes not handled, formatting issues)
Update process:
- Note issues encountered during generation
- Identify patterns that caused problems
- Update relevant files:
SKILL.md- Workflow, decision tree, quick reference updatestemplates/*.md- Template improvementsdiscovery.md- Detection logic updatestesting.md- New validation checks
- Document the improvement in commit message
Example self-improvement:
Issue: libFuzzer skill missing sanitizer flags table
Fix: Updated templates/fuzzer-skill.md to include ## Compiler Flags section
Example Usage
Full Discovery and Generation
User: "Generate skills from the testing handbook"
1. Locate handbook (check common locations, ask user, or clone)
2. Read discovery.md for methodology
3. Scan handbook at {handbook_path}/content/docs/
4. Build candidate list with types
5. Present plan to user
6. On approval, generate each skill using appropriate template
7. Validate generated skills
8. Update main README.md with generated skills table
9. Update README.md Skills Cross-Reference graph from Related Skills sections
10. Self-improve: note any template/discovery issues for future runs
11. Report results
Single Section Generation
User: "Create a skill for the libFuzzer section"
1. Read /testing-handbook/content/docs/fuzzing/c-cpp/10-libfuzzer/
2. Identify type: Fuzzer Skill
3. Read templates/fuzzer-skill.md
4. Extract content, apply template
5. Write to skills/libfuzzer/SKILL.md
6. Validate and report
Tips
Do:
- Always present plan before generating
- Use appropriate template for skill type
- Preserve code blocks exactly
- Validate after generation
Don't:
- Generate without user approval
- Skip fetching non-video external resources (use WebFetch)
- Fetch video URLs (YouTube, Vimeo - titles only)
- Include handbook images directly
- Skip validation step
- Exceed 500 lines per SKILL.md
For first-time use: Start with discovery.md to understand the handbook analysis process.
For template reference: See templates/ directory for skill type templates.
For validation: See testing.md for quality assurance methodology.
Vector Forge
Uses mutation testing to systematically identify gaps in test vector coverage, then generates new test vectors that close those gaps. Measures effectiveness by comparing mutation kill rates before and after.
When to Use
- Generating test vectors for cryptographic algorithms or protocols
- Evaluating how well existing test vectors cover an implementation
- Finding implementation code paths that no test vector exercises
- Creating Wycheproof-style cross-implementation test vectors
- Measuring the concrete coverage value of a test vector suite
When NOT to Use
- No implementations exist yet (need code to mutate)
- Single trivial implementation with no edge cases
- Testing application logic rather than algorithm implementations
- The algorithm has no public test vectors to compare against
Prerequisites
- trailmark installed — if
uv run trailmarkfails, run:uv tool install trailmark
Python snippets: uv run --with trailmark python - (a tool env is not importable)
- At least one implementation of the target algorithm in a
language with mutation testing support
- A test harness that consumes test vectors and exercises
the implementation
- A mutation testing framework for the target language
---
## Rationalizations to Reject
| Rationalization | Why It's Wrong | Required Action |
|-----------------|----------------|-----------------|
| "We have enough test vectors" | Mutation testing proves otherwise | Run the baseline first |
| "The implementation's own tests are sufficient" | Own tests often share blind spots with the impl | Cross-impl vectors catch different bugs |
| "FFI crates can be mutation tested at the binding layer" | Mutations to wrappers don't affect the underlying impl | Mutate the actual implementation language |
| "Timeouts mean the mutation was caught" | Timeouts are ambiguous — could be killed or alive | Resolve timeouts before drawing conclusions |
| "All mutants are equivalent" | Most aren't — verify by reading the mutation | Classify each escaped mutant individually |
| "Checking valid vectors is enough" | Permissive mutations survive without negative assertions | Assert rejection for every invalid vector |
| "Manual analysis is fine" | Manual analysis misses what tooling catches | Install and run the tools |
---
## Workflow Overview
Phase 1: Discovery → Find implementations to test ↓ Phase 2: Harness → Write/adapt test vector harness for each impl ↓ Phase 3: Baseline → Run mutation testing with existing vectors ↓ Phase 4: Escape Analysis → Classify escaped mutants by code path ↓ Phase 5: Vector Gen → Create test vectors targeting escapes ↓ Phase 6: Validation → Re-run mutation testing, compare before/after ↓ Output: Coverage Report + New Test Vectors
---
## Phase 1: Discovery
Find implementations of the target algorithm. Look for:
1. **Pure implementations** in high-level languages (Go, Rust, Python)
— these are the best mutation testing targets
2. **FFI wrapper crates** — identify these early so you don't waste
time mutating wrapper glue code
3. **Reference implementations** — useful for cross-verification but
may not be the best mutation targets
For each implementation, note:
- Language and mutation testing framework
- Whether it's pure code or FFI wrappers
- Existing test suite size and coverage
- Which API surface the test vectors will exercise
### Implementation Type Classification
| Type | Mutation Value | Example |
|------|---------------|---------|
| Pure implementation | High | zkcrypto/bls12_381 (Rust), gnark-crypto (Go) |
| FFI bindings to C/asm | Low at binding layer | blst Rust crate |
| C/C++ implementation | High (use Mull) | blst C library |
| Generated code | Medium (mutations may be equivalent) | gnark-crypto generated field arithmetic |
**Key insight:** If an implementation delegates to another language
via FFI, you must mutate the *underlying* implementation, not the
bindings. For C/C++ underneath Rust/Go/Python, use Mull or similar.
---
## Phase 2: Harness
For each implementation, create a test harness that:
1. Reads test vectors from JSON files (Wycheproof format recommended)
2. Exercises the implementation's API for each vector
3. Asserts **both acceptance and rejection**:
- Valid vectors: deserialization succeeds, output matches expected
- Invalid vectors: deserialization fails or verification rejects
4. Adds **roundtrip assertions** for valid deserialization vectors:
`serialize(deserialize(bytes)) == bytes`
5. Reports pass/fail per vector with test IDs
**Critical:** A harness that only checks valid vectors will miss all
permissive mutations (e.g., `&` → `|` in validation). See
[references/lessons-learned.md](references/lessons-learned.md) §7.
The harness must be runnable by the mutation testing framework.
For most frameworks this means:
- **Go:** A `_test.go` file in the same package as the implementation
- **Rust:** An integration test in `tests/` or inline `#[test]` functions
- **Python:** A pytest test file
- **C/C++:** A test binary linked against the implementation
### Harness Placement
The harness must live *inside the implementation's package* so the
mutation framework can see it. This usually means:
```bash
# Go: add test file to the package being mutated
cp wycheproof_test.go /path/to/impl/package/
# Rust: add integration test
cp wycheproof.rs /path/to/crate/tests/
# Python: add test to the test directory
cp test_wycheproof.py /path/to/package/tests/
Handling Existing Vectors
If the implementation already has test vectors:
- Run mutation testing with ONLY the existing vectors (baseline)
- Run mutation testing with ONLY your new vectors
- Run mutation testing with BOTH combined
- The delta between (1) and (3) shows the new vectors' value
Phase 3: Baseline
Run mutation testing with existing test vectors only.
Framework Selection
See references/mutation-frameworks.md for language-specific setup.
| Language | Framework | Command |
|---|---|---|
| Go | gremlins | gremlins unleash ./path/to/package |
| Rust | cargo-mutants | cargo mutants -j N --timeout T |
| Python | mutmut | mutmut run --paths-to-mutate src/ |
| C/C++ | Mull | mull-runner -test-framework=GoogleTest binary |
Parallelism
Always use parallel execution for large codebases:
cargo mutants -j 8(Rust, 8 parallel workers)gremlins unleash --timeout-coefficient 3(Go, increase timeouts)mutmut run --runner "pytest -x -q"(Python, fail-fast)
Recording Baseline Results
Capture these metrics per implementation:
| Metric | Description |
|---|---|
| Total mutants | Number of mutations generated |
| Killed | Mutants caught by tests |
| Survived/Lived | Mutants NOT caught (these are the targets) |
| Not covered | Code paths no test reaches at all |
| Timed out | Ambiguous — resolve before comparing |
| Efficacy % | Killed / (Killed + Survived) |
| Coverage % | (Total - Not covered) / Total |
Save the full mutation log for Phase 4 analysis.
Phase 4: Escape Analysis (Graph-Informed Triage)
Classify each escaped (survived + not covered) mutant using the Trailmark call graph for reachability and blast radius analysis.
This phase MUST use the genotoxic skill's triage methodology. The call graph transforms mutation results from a flat list of survived mutants into an actionable, prioritized set of vector targets.
Step 1: Build the Call Graph
Build a Trailmark code graph for each implementation before triaging mutations:
# Go
uv run trailmark analyze --language go --summary {targetDir}
# Rust
uv run trailmark analyze --language rust --summary {targetDir}
The graph provides:
- Caller chains — trace from public API entry points to mutated functions to determine reachability
- Cyclomatic complexity — prioritize high-CC functions
- Blast radius — functions with many callers have wider impact if their mutations survive
Step 2: Filter to Relevant Code
Mutation frameworks test the entire package. Filter results to only the files/functions that test vectors should exercise:
# Go (gremlins)
grep -E "(LIVED|NOT COVERED)" baseline.log \
| grep -E " at (relevant|files)" \
| sort
# Rust (cargo-mutants)
cat mutants.out/missed.txt | grep "src/relevant"
Step 3: Graph-Informed Classification
For each escaped mutant, map it to its containing function in the call graph and apply the genotoxic triage criteria:
| Graph Signal | Classification | Action |
|---|---|---|
| No callers in graph | False Positive | Dead code, skip |
| Only test callers | False Positive | Test infrastructure |
| Logging/display/formatting | False Positive | Cosmetic |
| Cross-package callers but NOT COVERED | Cross-Package Gap | See below |
| Reachable from public API, low CC | Missing Vector | Design targeted vector |
| Reachable from public API, high CC (>10) | Fuzzing Target | Both vector + fuzz harness |
| Validation/error-handling path | Negative Vector | Craft invalid input that triggers path |
| Optimization path (GLV, SIMD, batch) | Edge-Case Vector | Input that triggers optimization threshold |
|→^ after left shift (e.g. (t<<1) | carry) | Equivalent Mutant | Skip — bit 0 always 0, OR=XOR |
ct_eq &→| on Montgomery limbs | API-Unreachable | Needs library-internal tests, not vectors |
| Equivalent mutation (behavior unchanged) | False Positive | Skip |
Step 4: Identify Cross-Package Test Gaps
Critical pitfall: Mutation frameworks often only run tests within the same package as the mutation. For Go (gremlins) and Rust (cargo-mutants), this means:
- A mutation in
hash_to_curve/g2.goonly runs tests in thehash_to_curvepackage, NOT tests in the parentbls12381package that imports it - Functions that are fully exercised by cross-package tests will appear as NOT COVERED — these are false positives
- To confirm: check if the mutated function is called from a test in a different package that wouldn't be run
To resolve cross-package gaps:
- Add a thin test in the sub-package that calls through the same code path as the cross-package test
- Or run gremlins with
--test-pkg ./...(if supported) - Or document as a framework limitation in the report
Step 5: Prioritize by Security Impact
Using the call graph, rank surviving mutants by impact:
| Priority | Criteria | Example |
|---|---|---|
| P0 — Critical | Mutant weakens validation/equality/authentication | ct_eq: & → | makes equality permissive |
| P1 — High | Mutant in deserialization flag parsing | from_compressed: & → | accepts invalid flags |
| P2 — Medium | Mutant in field arithmetic internals | Fp::square: | → ^ corrupts computation |
| P3 — Low | Mutant in optimization path | phi endomorphism: only affects performance path |
| Skip | Formatting, display, equivalent mutation | Debug::fmt return value replacement |
Step 6: Group by Vector Strategy
Group escaped mutants by the code path they represent and the type of test vector needed:
Deserialization flag validation (P1):
- g1.rs:339,363-365,384 — from_compressed_unchecked flags
→ Need: valid-point-wrong-flag vectors
Field arithmetic (P2):
- fp.rs:371-376,406,635-643 — subtract_p, neg, square
→ Need: field arithmetic KATs with edge-case values
Optimization thresholds (P3):
- g1.go:68, g2.go:75 — GLV vs windowed multiplication
→ Need: scalar multiplication with large scalars
Cross-package (framework limitation):
- hash_to_curve/g2.go:242-278 — isogeny, sgn0
→ Document as false positive or add sub-package test
Each group becomes a target for new test vectors in Phase 5.
Phase 5: Vector Generation
For each escaped code path group, design test vectors that force execution through that path.
Vector Design Patterns
| Code Path Type | Vector Strategy |
|---|---|
| Point deserialization | Malformed points: wrong length, invalid field elements, off-curve, wrong subgroup, identity point |
| Signature verification | Valid sig + all single-bit corruptions of sig, pk, msg |
| Hash-to-curve | Known answer tests (KATs) with edge-case inputs: empty, single byte, max length |
| Aggregate operations | 1 signer, many signers, duplicate signers, mixed valid/invalid |
| Error handling | Every error path should have a vector that triggers it |
| Arithmetic edge cases | Zero, one, field modulus - 1, points at infinity |
| Serialization flags | Every valid flag combination + every invalid flag combination |
| Roundtrip integrity | For every valid deser vector, assert serialize(deserialize(b)) == b |
| Carry/reduction faults | Reimplement at reduced limb widths, inject faults, extract distinguishing inputs |
Single-Fault Negative Vectors
Each negative vector should have exactly one defect with everything else valid — this isolates which validation check is being tested. See references/vector-patterns.md for per-flag construction examples.
Fault Simulation (Limb-Width Reimplementation)
When mutation testing only applies local operator swaps, deeper architectural bugs (carry propagation, reduction overflow) go untested. To close this gap, reimplement the target algorithm at reduced limb widths (8, 16, 25, 32 bits) and deliberately inject faults — then generate vectors that catch them.
See references/fault-simulation.md for the full methodology: limb-width selection, fault injection catalog, vector extraction, and validation workflow.
Cross-Implementation Verification
Every new test vector MUST be verified against at least two independent implementations before being added to the suite:
- Generate the vector using implementation A
- Verify with implementation B (different codebase, ideally different language)
- If B disagrees, investigate — one implementation has a bug
Vector Format
Use Wycheproof JSON format (algorithm, testGroups[].tests[]
with tcId, comment, result, flags). See
references/vector-patterns.md
for the full schema.
Wycheproof contributions: Use Wycheproof's vectorgen tool rather
than formatting vector files directly. Supply the generated changes as an
envelope. The vectorgen tool can add, update, or replace vectors while
handling tcId assignment, test counts, canonical formatting, and schema
validation. Go-based generators can avoid the vectorgen CLI tool and instead
call the programmatic github.com/c2sp/wycheproof/vectorgen API.
See references/lessons-learned.md §14 and the upstream vectorgen guide for the current workflow and commands.
Phase 6: Validation
Re-run mutation testing with the new test vectors included.
Tip: Use per-file mutation testing for fast iteration during vector development (see references/lessons-learned.md §12). Only run full-crate tests for the final comparison.
Before/After Comparison
| Metric | Baseline | With New Vectors | Delta |
|---|---|---|---|
| Killed | X | Y | Y - X |
| Survived | A | B | A - B (should decrease) |
| Not Covered | C | D | C - D (should decrease) |
| Efficacy % | E% | F% | F - E |
Success Criteria
Vectors have both retroactive value (killing mutants in existing code) and proactive value (catching bugs in future implementations). Generate both kinds — boundary-condition vectors may not improve kill rates in mature libraries but will catch bugs in new implementations. See references/lessons-learned.md §13.
Retroactive (measurable): previously survived/uncovered mutants become killed, no regressions.
If kill rates don't change: the implementation's own tests likely already cover those paths. The vectors still add cross-implementation verification value. Document which case applies.
Output Format
Write VECTOR_FORGE_REPORT.md covering: target algorithm,
implementations tested, baseline results, escape analysis,
new vectors generated, after results, before/after delta, and
conclusions. See
references/report-template.md
for the full template.
Quality Checklist
Before delivering:
- At least one pure implementation mutation-tested (not just FFI wrappers)
- Baseline run completed with existing vectors
- Trailmark call graph built for each implementation
- All escaped mutants triaged using graph-informed classification
- Cross-package false positives identified and documented
- Security-critical mutations (ct_eq, validation, auth) prioritized as P0/P1
- Fault simulation and mutation-derived vectors cross-verified against 2+ implementations
- After run completed with new vectors included
- Before/after delta computed and explained
- Report written to
VECTOR_FORGE_REPORT.md - New test vectors saved in standard format (Wycheproof JSON)
Integration
| Skill | Relationship |
|---|---|
| genotoxic (required for Phase 4) | Provides graph-informed triage — call graph cuts actionable mutants by 30-50% |
| mutation-testing (mewt/muton) | Use for Solidity; Vector Forge is language-agnostic |
| property-based-testing | Better than hand-crafted vectors for bitwise mutations in field arithmetic |
| testing-handbook-skills (fuzzing) | Functions with CC > 10 and surviving mutants need both vectors and fuzz harnesses |
Supporting Documentation
- references/mutation-frameworks.md - Language-specific mutation testing framework setup
- references/vector-patterns.md - Common test vector patterns for cryptographic primitives
- references/fault-simulation.md - Limb-width reimplementation for carry, reduction, and overflow faults
- references/report-template.md - Full markdown template for the Vector Forge report
- references/lessons-learned.md - BLS12-381 case study: FFI kill rates, timeout masking, cross-package false positives, bitwise mutation gaps, and security-critical priorities
Wycheproof
Wycheproof is an extensive collection of test vectors designed to verify the correctness of cryptographic implementations and test against known attacks. Originally developed by Google, it is now a community-managed project where contributors can add test vectors for specific cryptographic constructions.
Background
Key Concepts
| Concept | Description |
|---|---|
| Test vector | Input/output pair for validating crypto implementation correctness |
| Test group | Collection of test vectors sharing attributes (key size, IV size, curve) |
| Result flag | Indicates if test should pass (valid), fail (invalid), or is acceptable |
| Edge case testing | Testing for known vulnerabilities and attack patterns |
Why This Matters
Cryptographic implementations are notoriously difficult to get right. Even small bugs can:
- Expose private keys
- Allow signature forgery
- Enable message decryption
- Create consensus problems when different implementations accept/reject the same inputs
Wycheproof has found vulnerabilities in major libraries including OpenJDK's SHA1withDSA, Bouncy Castle's ECDHC, and the elliptic npm package.
When to Use
Apply Wycheproof when:
- Testing cryptographic implementations (AES-GCM, ECDSA, ECDH, RSA, etc.)
- Validating that crypto code handles edge cases correctly
- Verifying implementations against known attack vectors
- Setting up CI/CD for cryptographic libraries
- Auditing third-party crypto code for correctness
Consider alternatives when:
- Testing for timing side-channels (use constant-time testing tools instead)
- Finding new unknown bugs (use fuzzing instead)
- Testing custom/experimental cryptographic algorithms (Wycheproof only covers established algorithms)
Quick Reference
| Scenario | Recommended Approach | Notes |
|---|---|---|
| AES-GCM implementation | Use aes_gcm_test.json | 316 test vectors across 44 test groups |
| ECDSA verification | Use ecdsa_*_test.json for specific curves | Tests signature malleability, DER encoding |
| ECDH key exchange | Use ecdh_*_test.json | Tests invalid curve attacks |
| RSA signatures | Use rsa_*_test.json | Tests padding oracle attacks |
| ChaCha20-Poly1305 | Use chacha20_poly1305_test.json | Tests AEAD implementation |
Testing Workflow
Phase 1: Setup Phase 2: Parse Test Vectors
┌─────────────────┐ ┌─────────────────┐
│ Add Wycheproof │ → │ Load JSON file │
│ as submodule │ │ Filter by params│
└─────────────────┘ └─────────────────┘
↓ ↓
Phase 4: CI Integration Phase 3: Write Harness
┌─────────────────┐ ┌─────────────────┐
│ Auto-update │ ← │ Test valid & │
│ test vectors │ │ invalid cases │
└─────────────────┘ └─────────────────┘
Repository Structure
The Wycheproof repository is organized as follows:
┣ 📜 README.md : Project overview
┣ 📂 doc : Documentation
┣ 📂 java : Java JCE interface testing harness
┣ 📂 javascript : JavaScript testing harness
┣ 📂 schemas : Test vector schemas
┣ 📂 testvectors : Test vectors
┗ 📂 testvectors_v1 : Updated test vectors (more detailed)
The essential folders are testvectors and testvectors_v1. While both contain similar files, testvectors_v1 includes more detailed information and is recommended for new integrations.
Supported Algorithms
Wycheproof provides test vectors for a wide range of cryptographic algorithms:
| Category | Algorithms |
|---|---|
| Symmetric Encryption | AES-GCM, AES-EAX, ChaCha20-Poly1305 |
| Signatures | ECDSA, EdDSA, RSA-PSS, RSA-PKCS1 |
| Key Exchange | ECDH, X25519, X448 |
| Hashing | HMAC, HKDF |
| Curves | secp256k1, secp256r1, secp384r1, secp521r1, ed25519, ed448 |
Test File Structure
Each JSON test file tests a specific cryptographic construction. All test files share common attributes:
"algorithm" : The name of the algorithm tested
"schema" : The JSON schema (found in schemas folder)
"generatorVersion" : The version number
"numberOfTests" : The total number of test vectors in this file
"header" : Detailed description of test vectors
"notes" : In-depth explanation of flags in test vectors
"testGroups" : Array of one or multiple test groups
Test Groups
Test groups group sets of tests based on shared attributes such as:
- Key sizes
- IV sizes
- Public keys
- Curves
This classification allows extracting tests that meet specific criteria relevant to the construction being tested.
Test Vector Attributes
Shared Attributes
All test vectors contain four common fields:
- tcId: Unique identifier for the test vector within a file
- comment: Additional information about the test case
- flags: Descriptions of specific test case types and potential dangers (referenced in
notesfield) - result: Expected outcome of the test
The result field can take three values:
| Result | Meaning |
|---|---|
| valid | Test case should succeed |
| acceptable | Test case is allowed to succeed but contains non-ideal attributes |
| invalid | Test case should fail |
Unique Attributes
Unique attributes are specific to the algorithm being tested:
| Algorithm | Unique Attributes |
|---|---|
| AES-GCM | key, iv, aad, msg, ct, tag |
| ECDH secp256k1 | public, private, shared |
| ECDSA | msg, sig, result |
| EdDSA | msg, sig, pk |
Implementation Guide
Phase 1: Add Wycheproof to Your Project
Option 1: Git Submodule (Recommended)
Adding Wycheproof as a git submodule ensures automatic updates:
git submodule add https://github.com/C2SP/wycheproof.git
Option 2: Fetch Specific Test Vectors
If submodules aren't possible, fetch specific JSON files:
#!/bin/bash
TMP_WYCHEPROOF_FOLDER=".wycheproof/"
TEST_VECTORS=('aes_gcm_test.json' 'aes_eax_test.json')
BASE_URL="https://raw.githubusercontent.com/C2SP/wycheproof/master/testvectors_v1/"
# Create wycheproof folder
mkdir -p $TMP_WYCHEPROOF_FOLDER
# Request all test vector files if they don't exist
for i in "${TEST_VECTORS[@]}"; do
if [ ! -f "${TMP_WYCHEPROOF_FOLDER}${i}" ]; then
curl -o "${TMP_WYCHEPROOF_FOLDER}${i}" "${BASE_URL}${i}"
if [ $? -ne 0 ]; then
echo "Failed to download ${i}"
exit 1
fi
fi
done
Phase 2: Parse Test Vectors
Identify the test file for your algorithm and parse the JSON:
Python Example:
import json
def load_wycheproof_test_vectors(path: str):
testVectors = []
try:
with open(path, "r") as f:
wycheproof_json = json.loads(f.read())
except FileNotFoundError:
print(f"No Wycheproof file found at: {path}")
return testVectors
# Attributes that need hex-to-bytes conversion
convert_attr = {"key", "aad", "iv", "msg", "ct", "tag"}
for testGroup in wycheproof_json["testGroups"]:
# Filter test groups based on implementation constraints
if testGroup["ivSize"] < 64 or testGroup["ivSize"] > 1024:
continue
for tv in testGroup["tests"]:
# Convert hex strings to bytes
for attr in convert_attr:
if attr in tv:
tv[attr] = bytes.fromhex(tv[attr])
testVectors.append(tv)
return testVectors
JavaScript Example:
const fs = require('fs').promises;
async function loadWycheproofTestVectors(path) {
const tests = [];
try {
const fileContent = await fs.readFile(path);
const data = JSON.parse(fileContent.toString());
data.testGroups.forEach(testGroup => {
testGroup.tests.forEach(test => {
// Add shared test group properties to each test
test['pk'] = testGroup.publicKey.pk;
tests.push(test);
});
});
} catch (err) {
console.error('Error reading or parsing file:', err);
throw err;
}
return tests;
}
Phase 3: Write Testing Harness
Create test functions that handle both valid and invalid test cases.
Python/pytest Example:
import pytest
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
tvs = load_wycheproof_test_vectors("wycheproof/testvectors_v1/aes_gcm_test.json")
@pytest.mark.parametrize("tv", tvs, ids=[str(tv['tcId']) for tv in tvs])
def test_encryption(tv):
try:
aesgcm = AESGCM(tv['key'])
ct = aesgcm.encrypt(tv['iv'], tv['msg'], tv['aad'])
except ValueError as e:
# Implementation raised error - verify test was expected to fail
assert tv['result'] != 'valid', tv['comment']
return
if tv['result'] == 'valid':
assert ct[:-16] == tv['ct'], f"Ciphertext mismatch: {tv['comment']}"
assert ct[-16:] == tv['tag'], f"Tag mismatch: {tv['comment']}"
elif tv['result'] == 'invalid' or tv['result'] == 'acceptable':
assert ct[:-16] != tv['ct'] or ct[-16:] != tv['tag']
@pytest.mark.parametrize("tv", tvs, ids=[str(tv['tcId']) for tv in tvs])
def test_decryption(tv):
try:
aesgcm = AESGCM(tv['key'])
decrypted_msg = aesgcm.decrypt(tv['iv'], tv['ct'] + tv['tag'], tv['aad'])
except ValueError:
assert tv['result'] != 'valid', tv['comment']
return
except InvalidTag:
assert tv['result'] != 'valid', tv['comment']
assert 'ModifiedTag' in tv['flags'], f"Expected 'ModifiedTag' flag: {tv['comment']}"
return
assert tv['result'] == 'valid', f"No invalid test case should pass: {tv['comment']}"
assert decrypted_msg == tv['msg'], f"Decryption mismatch: {tv['comment']}"
JavaScript/Mocha Example:
const assert = require('assert');
function testFactory(tcId, tests) {
it(`[${tcId + 1}] ${tests[tcId].comment}`, function () {
const test = tests[tcId];
const ed25519 = new eddsa('ed25519');
const key = ed25519.keyFromPublic(toArray(test.pk, 'hex'));
let sig;
if (test.result === 'valid') {
sig = key.verify(test.msg, test.sig);
assert.equal(sig, true, `[${test.tcId}] ${test.comment}`);
} else if (test.result === 'invalid') {
try {
sig = key.verify(test.msg, test.sig);
} catch (err) {
// Point could not be decoded
sig = false;
}
assert.equal(sig, false, `[${test.tcId}] ${test.comment}`);
}
});
}
// Generate tests for all test vectors
for (var tcId = 0; tcId < tests.length; tcId++) {
testFactory(tcId, tests);
}
Phase 4: CI Integration
Ensure test vectors stay up to date by:
- Using git submodules: Update submodule in CI before running tests
- Fetching latest vectors: Run fetch script before test execution
- Scheduled updates: Set up weekly/monthly updates to catch new test vectors
Common Vulnerabilities Detected
Wycheproof test vectors are designed to catch specific vulnerability patterns:
| Vulnerability | Description | Affected Algorithms | Example CVE |
|---|---|---|---|
| Signature malleability | Multiple valid signatures for same message | ECDSA, EdDSA | CVE-2024-42459 |
| Invalid DER encoding | Accepting non-canonical DER signatures | ECDSA | CVE-2024-42460, CVE-2024-42461 |
| Invalid curve attacks | ECDH with invalid curve points | ECDH | Common in many libraries |
| Padding oracle | Timing leaks in padding validation | RSA-PKCS1 | Historical OpenSSL issues |
| Tag forgery | Accepting modified authentication tags | AES-GCM, ChaCha20-Poly1305 | Various implementations |
Signature Malleability: Deep Dive
Problem: Implementations that don't validate signature encoding can accept multiple valid signatures for the same message.
Example (EdDSA): Appending or removing zeros from signature:
Valid signature: ...6a5c51eb6f946b30d
Invalid signature: ...6a5c51eb6f946b30d0000 (should be rejected)
How to detect:
# Add signature length check
if len(sig) != 128: # EdDSA signatures must be exactly 64 bytes (128 hex chars)
return False
Impact: Can lead to consensus problems when different implementations accept/reject the same signatures.
Related Wycheproof tests:
- EdDSA: tcId 37 - "removing 0 byte from signature"
- ECDSA: tcId 06 - "Legacy: ASN encoding of r misses leading 0"
Case Study: Elliptic npm Package
This case study demonstrates how Wycheproof found three CVEs in the popular elliptic npm package (3000+ dependents, millions of weekly downloads).
Overview
The elliptic library is an elliptic-curve cryptography library written in JavaScript, supporting ECDH, ECDSA, and EdDSA. Using Wycheproof test vectors on version 6.5.6 revealed multiple vulnerabilities:
- CVE-2024-42459: EdDSA signature malleability (appending/removing zeros)
- CVE-2024-42460: ECDSA DER encoding - invalid bit placement
- CVE-2024-42461: ECDSA DER encoding - leading zero in length field
Methodology
- Identify supported curves: ed25519 for EdDSA
- Find test vectors:
testvectors_v1/ed25519_test.json - Parse test vectors: Load JSON and extract tests
- Write test harness: Create parameterized tests
- Run tests: Identify failures
- Analyze root causes: Examine implementation code
- Propose fixes: Add validation checks
Key Findings
EdDSA Issue (CVE-2024-42459):
- Missing signature length validation
- Allowed trailing zeros in signatures
- Fix: Add
if(sig.length !== 128) return false;
ECDSA Issue 1 (CVE-2024-42460):
- Missing check for first bit being zero in DER-encoded r and s values
- Fix: Add
if ((data[p.place] & 128) !== 0) return false;
ECDSA Issue 2 (CVE-2024-42461):
- DER length field accepted leading zeros
- Fix: Add
if(buf[p.place] === 0x00) return false;
Impact
All three vulnerabilities allowed multiple valid signatures for a single message, leading to consensus problems across implementations.
Lessons learned:
- Wycheproof catches subtle encoding bugs
- Reusable test harnesses pay dividends
- Test vector comments and flags help diagnose issues
- Even popular libraries benefit from systematic test vector validation
Advanced Usage
Tips and Tricks
| Tip | Why It Helps |
|---|---|
| Filter test groups by parameters | Focus on test vectors relevant to your implementation constraints |
| Use test vector flags | Understand specific vulnerability patterns being tested |
Check the notes field | Get detailed explanations of flag meanings |
| Test both encrypt/decrypt and sign/verify | Ensure bidirectional correctness |
| Run tests in CI | Catch regressions and benefit from new test vectors |
| Use parameterized tests | Get clear failure messages with tcId and comment |
Common Mistakes
| Mistake | Why It's Wrong | Correct Approach |
|---|---|---|
| Only testing valid cases | Misses vulnerabilities where invalid inputs are accepted | Test all result types: valid, invalid, acceptable |
| Ignoring "acceptable" result | Implementation might have subtle bugs | Treat acceptable as warnings worth investigating |
| Not filtering test groups | Wastes time on unsupported parameters | Filter by keySize, ivSize, etc. based on your implementation |
| Not updating test vectors | Miss new vulnerability patterns | Use submodules or scheduled fetches |
| Testing only one direction | Encrypt/sign might work but decrypt/verify fails | Test both operations |
Related Skills
Tool Skills
| Skill | Primary Use in Wycheproof Testing |
|---|---|
| pytest | Python testing framework for parameterized tests |
| mocha | JavaScript testing framework for test generation |
| constant-time-testing | Complement Wycheproof with timing side-channel testing |
| cryptofuzz | Fuzz-based crypto testing to find additional bugs |
Technique Skills
| Skill | When to Apply |
|---|---|
| coverage-analysis | Ensure test vectors cover all code paths in crypto implementation |
| property-based-testing | Test mathematical properties (e.g., encrypt/decrypt round-trip) |
| fuzz-harness-writing | Create harnesses for crypto parsers (complements Wycheproof) |
Related Domain Skills
| Skill | Relationship |
|---|---|
| crypto-testing | Wycheproof is a key tool in comprehensive crypto testing methodology |
| fuzzing | Use fuzzing to find bugs Wycheproof doesn't cover (new edge cases) |
Skill Dependency Map
┌─────────────────────┐
│ wycheproof │
│ (this skill) │
└──────────┬──────────┘
│
┌───────────────────┼───────────────────┐
│ │ │
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ pytest/mocha │ │ constant-time │ │ cryptofuzz │
│ (test framework)│ │ testing │ │ (fuzzing) │
└────────┬────────┘ └────────┬────────┘ └────────┬────────┘
│ │ │
└───────────────────┼───────────────────┘
│
▼
┌──────────────────────────┐
│ Technique Skills │
│ coverage, harness, PBT │
└──────────────────────────┘
Resources
Official Repository
The official repository contains:
- All test vectors in
testvectors/andtestvectors_v1/ - JSON schemas in
schemas/ - Reference implementations in Java and JavaScript
- Documentation in
doc/
Real-World Examples
The pycryptodome library integrates Wycheproof test vectors in their test suite, demonstrating best practices for Python crypto implementations.
Community Resources
- C2SP Community - Cryptographic specifications and standards community maintaining Wycheproof
- Wycheproof issues tracker - Report bugs in test vectors or suggest new constructions
Summary
Wycheproof is an essential tool for validating cryptographic implementations against known attack vectors and edge cases. By integrating Wycheproof test vectors into your testing workflow:
- Catch subtle encoding and validation bugs
- Prevent signature malleability issues
- Ensure consistent behavior across implementations
- Benefit from community-contributed test vectors
- Protect against known cryptographic vulnerabilities
The investment in writing a reusable testing harness pays dividends through continuous validation as new test vectors are added to the Wycheproof repository.
Sponsors
Details
- Marketplace
- marslanalvi/SkillsStack
- Source
- plugins/security/fuzzing
- Category
- Security
- Keywords
- security, fuzzing
More from marslanalvi/SkillsStack
All 45- security-offensive-securityv1.0.0
- ai-agent-engineeringv1.0.0
- ai-coding-assistant-workflowv1.0.0
- ai-llm-applicationsv1.0.0
- ai-machine-learningv1.0.0
- ai-skill-authoringv1.0.0
- business-legal-industryv1.0.0
- business-operations-financev1.0.0
- business-strategy-salesv1.0.0
- data-data-engineering-analyticsv1.0.0
- data-researchv1.0.0
- design-accessibilityv1.0.0
- design-media-creativev1.0.0
- design-ui-uxv1.0.0
- devops-ci-cdv1.0.0
- devops-cloudv1.0.0
- devops-containers-kubernetesv1.0.0
- devops-infrastructure-networkingv1.0.0
- devops-observability-srev1.0.0
- engineering-architecture-apiv1.0.0
- engineering-code-review-qualityv1.0.0
- engineering-databasesv1.0.0
- engineering-debuggingv1.0.0
- engineering-dev-workflowv1.0.0