Summary
Adds consensus measurement tools to Claude using Fleiss' kappa and bootstrap confidence intervals to check if AI models agree with themselves or each other. The eight MCP tools let you run multi-model evaluations across Bedrock, OpenAI, and Gemini, generate statistical reports, compare runs over time, and estimate costs before executing. The self-consistency mode is handy because it uses MCP Sampling to test the host model without external API keys. You'd reach for this when you need statistically rigorous validation that an AI is giving consistent answers, especially for high-stakes applications where agreement matters more than speed. Includes schema validation and AI-powered schema suggestion from your data.
Fastest browser for AI agents to run web automation tasks, always free.
Download Free life-time →
belt cli automatically finds the best tools and skills for your agent. image, video, music, tts...
one prompt install →
Free an open-source skills that make AI coding agents easier to understand, verify, and control.
Download for free →
Connect your Claude agent to live crypto prices and trading routes via 1inch
Get the MCP →
Notes, actions and memory. Without a meeting bot. First month 100% off.
Download for free →ConKurrence
Multi-model rating for AI evaluation: measures agreement among model raters (Fleiss' κ, Kendall's W) to find contested items for expert review.
What agreement tells you: where your raters and your criteria are consistent, and where they are not — which items are contested, and whether disagreement comes from the criteria or the raters.
What it does not tell you: that the raters are right. Agreement is reliability, not validity. Models can agree confidently on a wrong answer. Use ConKurrence to decide where an expert's judgment is needed and to diagnose your evaluation criteria — not to replace expert-labelled ground truth.
Status: maintenance mode. ConKurrence is developed by JE Vectors LLC as the evaluation instrument behind Inqura. It is not accepting purchases and carries no support commitment.
Install
npm install -g conkurrence
MCP Server
Use ConKurrence as an MCP server in Claude Desktop or any MCP-compatible client:
npx conkurrence mcp
Claude Desktop Configuration
Add to your claude_desktop_config.json:
{
"mcpServers": {
"conkurrence": {
"command": "npx",
"args": ["-y", "conkurrence", "mcp"]
}
}
}
Claude Code Plugin
/plugin marketplace add AlligatorC0der/conkurrence
Features
- Multi-model evaluation — Run your schema against Bedrock, OpenAI, and Gemini models simultaneously
- Agreement statistics — Fleiss' kappa with bootstrap confidence intervals; Kendall's W against expert-labelled anchor items
- Self-consistency mode — No API keys needed; uses the host model via MCP Sampling
- Schema suggestion — AI-powered schema design from your data
- Trend tracking — Compare runs over time, detect agreement degradation
- Cost estimation — Know the cost before running
MCP Tools
| Tool | Description |
|---|---|
conkurrence_run | Execute an evaluation across multiple AI raters |
conkurrence_report | Generate a detailed markdown report |
conkurrence_compare | Side-by-side comparison of two runs |
conkurrence_trend | Track agreement over multiple runs |
conkurrence_suggest | AI-powered schema suggestion from your data |
conkurrence_validate_schema | Validate a schema before running |
conkurrence_estimate | Estimate cost and token usage |
Links
- Homepage: conkurrence.com
- npm: npmjs.com/package/conkurrence
- Terms of Service: app.conkurrence.com/terms
- Privacy Policy: app.conkurrence.com/privacy
License
BUSL-1.1 — Business Source License 1.1
Featured
Fastest browser for AI agents to run web automation tasks, always free.
Download Free life-time →belt cli automatically finds the best tools and skills for your agent. image, video, music, tts...
one prompt install →Free an open-source skills that make AI coding agents easier to understand, verify, and control.
Download for free →Connect your Claude agent to live crypto prices and trading routes via 1inch
Get the MCP →Notes, actions and memory. Without a meeting bot. First month 100% off.
Download for free →