Score and compare images using vision LLMs as judges. YAML-defined criteria presets for 11 use cases (text-to-image, pho

GitHub stars
414

Install

/plugin install vision-bench@glebis-skills

Run it inside Claude Code after adding the marketplace with /plugin marketplace add glebis/claude-skills.

Contents

Discovered from source

Skills 1

Vision Bench — LLM Image Evaluation

Compare images by scoring them with one or more vision LLM judges against structured rubric criteria.

Quick Start

# Install dependencies
pip install pyyaml openai anthropic mistralai

# Score a single image
python bench.py image.png --criteria photorealism --judge gemini-3.5-flash-lite

# Compare two AI-generated images
python bench.py img_a.png img_b.png \
  --criteria text_to_image \
  --prompt "a fox in a snowy forest" \
  --judge gpt-4o

# Multi-judge consensus
python bench.py img.png \
  --criteria portrait \
  --judges gpt-4o gemini-3.5-flash-lite claude-opus-4-5-20251022

# OpenRouter models (any vision-capable model)
python bench.py img_a.png img_b.png \
  --criteria artistic_style \
  --judges "openrouter/meta-llama/llama-4-maverick" "openrouter/mistralai/pixtral-large-2411"

# List all presets
python bench.py --list-presets

# Save report to file
python bench.py img.png --criteria chart_analysis --save report.md

Presets

PresetUse Case
text_to_imageCompare AI image generators (Midjourney, DALL-E, Flux)
photorealismHow convincingly an image looks like a photo
artistic_styleStyle consistency, composition, color harmony
portraitAI-generated portrait quality and realism
product_photoE-commerce product image quality
document_ocrDocument text extraction and layout understanding
chart_analysisChart and data visualization comprehension
invoiceFinancial document field extraction accuracy
ui_screenshotApp/web screenshot understanding
scientificScientific/medical image accuracy
alt_textAccessibility image description quality

Custom criteria: pass any .yaml file as --criteria path/to/my.yaml.

Judge Providers

PrefixProviderExample
gpt-, o1, o3, o4OpenAIgpt-4o
claude-Anthropicclaude-sonnet-4-5-20251022
gemini-Google Geminigemini-3.5-flash-lite
pixtral-, mistral-, ministral-Mistralpixtral-12b-2409
openrouter/OpenRouter (any model)openrouter/meta-llama/llama-4-maverick

API Keys

Keys are loaded from secrets.enc.yaml (SOPS + age encrypted) with fallback to environment variables.

Supported keys: OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, OPENROUTER_API_KEY

To encrypt your own keys:

sops --config .sops.yaml --encrypt --input-type yaml --output-type yaml secrets.yaml > secrets.enc.yaml

Output Formats

--output markdown (default) · --output json · --output table

Files

  • bench.py — CLI entry point
  • judge.py — Multi-provider LLM judge logic
  • report.py — Report generation
  • vault.py — SOPS secrets decryption
  • criteria/ — 11 YAML preset files
  • .sops.yaml — Age key config for encryption
  • secrets.enc.yaml — Encrypted API keys

Use --judge or --judges to select another model explicitly. The Gemini path sends one image/text turn with JSON output and no tools, history, thinking or sampling settings. The 3.5 Flash-Lite default requires a live image/JSON smoke test and rubric comparison before relying on historical score comparability; offline request tests do not establish model quality.

Sponsors

Details

Marketplace
glebis/claude-skills
Source
vision-bench
Category
Community

More from glebis/claude-skills

All 54