Universal LLM API client for 174 providers — chat, streaming, tools, embeddings, search, OCR, plus an OpenAI-compatible proxy and an MCP server.

GitHub stars
266

Install

/plugin install liter-llm@liter-llm

Run it inside Claude Code after adding the marketplace with /plugin marketplace add xberg-io/liter-llm.

Contents

Discovered from source

Skills 7

Calling LLMs

Build a ChatCompletionRequest and send it with client.chat(request). Create the client with create_client(...). The model string is provider/model; the prefix selects the backend.

import asyncio, json, os
from liter_llm import create_client
from liter_llm._internal_bindings import ChatCompletionRequest

async def main() -> None:
    client = create_client(api_key=os.environ["OPENAI_API_KEY"])
    request = ChatCompletionRequest.from_json(json.dumps({
        "model": "openai/gpt-4o",
        "messages": [
            {"role": "system", "content": "You are concise."},
            {"role": "user", "content": "Name three Rust crates for HTTP."},
        ],
    }))
    response = await client.chat(request)
    print(response.choices[0].message.content)

asyncio.run(main())

Provider routing

The model string's prefix selects the provider; build a request per backend:

ChatCompletionRequest.from_json('{"model":"anthropic/claude-sonnet-4-20250514","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"gemini/gemini-2.0-flash","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"groq/llama3-70b","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"mistral/mistral-large-latest","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"bedrock/anthropic.claude-v2","messages":[...]}')

Set a full provider/model as model_hint at construction to select the client provider and credential source. Requests without a recognized prefix fall back to that client provider:

client = create_client(api_key=os.environ["OPENAI_API_KEY"], model_hint="openai/gpt-4o")
# the request model can now omit the provider prefix:
request = ChatCompletionRequest.from_json('{"model":"gpt-4o","messages":[...]}')
await client.chat(request)  # routes to OpenAI

Notes

  • Keys come from env vars (OPENAI_API_KEY, ANTHROPIC_API_KEY, …); never hardcode them.
  • Without a recognized prefix, requests fall back to the client's configured provider; a client without model_hint defaults to OpenAI.
  • Environment credentials are selected when constructing the client, not independently for every request. Use a separate client with matching model_hint and credentials for each provider. A prefix changes routing but does not switch the client's API key.
  • Python errors are typed exceptions exported from liter_llm: AuthenticationError, RateLimitedError, BadRequestError, ContextWindowExceededError, ContentPolicyError, NotFoundError, ServerError, ServiceUnavailableError, LiterLlmTimeoutError, BudgetExceededError — all subclasses of LiterLlmError.

Embeddings and Search

liter-llm exposes embeddings, web search (12 providers), OCR (4 providers), and reranking through the same provider/model routing convention.

Embeddings

import asyncio, os
from liter_llm import create_client
from liter_llm._internal_bindings import EmbeddingRequest

async def main() -> None:
    client = create_client(api_key=os.environ["OPENAI_API_KEY"])
    request = EmbeddingRequest.from_json(
        '{"model":"openai/text-embedding-3-small","input":["first document","second document"]}'
    )
    response = await client.embed(request)
    for item in response.data:
        print(len(item.embedding))

asyncio.run(main())

Many embedding models support dimension selection and base64 output; set dimensions / encoding_format in the request where the provider allows it.

Web search (12 providers)

from liter_llm._internal_bindings import SearchRequest

client = create_client(api_key=os.environ["BRAVE_API_KEY"])
request = SearchRequest.from_json(
    '{"model":"brave/web-search","query":"What is the Rust programming language?","max_results":5}'
)
response = await client.search(request)
for result in response.results:
    print(result.title, result.url)

OCR (4 providers)

from liter_llm._internal_bindings import OcrRequest

client = create_client(api_key=os.environ["MISTRAL_API_KEY"])
request = OcrRequest.from_json(
    '{"model":"mistral/mistral-ocr-latest",'
    '"document":{"type":"document_url","url":"https://example.com/invoice.pdf"}}'
)
response = await client.ocr(request)
for page in response.pages:
    print(page.index, page.markdown[:100])

Reranking

Build a RerankRequest (model, query, documents) and call client.rerank(request) to score and order candidate documents against a query for retrieval pipelines — combine it with embed for hybrid retrieval. Each result carries index and relevance_score. Routing follows the same provider/model convention.

Notes

  • Search and OCR providers each need their own API key (e.g. BRAVE_API_KEY, MISTRAL_API_KEY); read them from env vars.
  • See the upstream provider reference for the full list of the 12 search and 4 OCR backends and their model identifiers.

Running the Proxy

liter-llm api is a drop-in OpenAI-compatible gateway: 22 REST endpoints that route to 174 providers, with multi-tenant virtual keys, rate limits, budgets, and cost tracking.

Start it

liter-llm api --config liter-llm-proxy.toml

The proxy auto-discovers liter-llm-proxy.toml in the current directory.

Configuration

[server]
host = "0.0.0.0"
port = 4000

[general]
master_key = "${LITER_LLM_MASTER_KEY}"

# Each [[models]] entry maps a routable name to a provider/model and its key.
[[models]]
name = "gpt-4o"
provider_model = "openai/gpt-4o"
api_key = "${OPENAI_API_KEY}"

[[models]]
name = "claude-sonnet"
provider_model = "anthropic/claude-sonnet-4-20250514"
api_key = "${ANTHROPIC_API_KEY}"

# Virtual keys scope which configured model names a caller may use.
[[keys]]
key = "sk-team-frontend"
models = ["gpt-4o", "claude-sonnet"]
rpm = 60
tpm = 100000
budget_limit = 50.0

${ENV_VAR} interpolation keeps secrets out of the file.

Call it like OpenAI

curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-team-frontend" \
  -d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "Hello"}]}'

Notes

  • Virtual keys scope which models a caller may use and carry their own RPM/TPM and budget limits; the master_key administers them.
  • The OpenAPI 3.1 spec is served at /openapi.json.
  • Endpoints cover chat, embeddings, images, audio, moderations, files, batches, responses, and model listing.
  • Docker: docker run -p 4000:4000 -e LITER_LLM_MASTER_KEY=sk-key ghcr.io/xberg-io/liter-llm.

Streaming Responses

Use chat_stream(...) to receive tokens as they are produced instead of waiting for the full completion. The proxy streams over SSE; bindings expose async iterators.

Python

import asyncio, os
from liter_llm import create_client
from liter_llm._internal_bindings import ChatCompletionRequest

async def main() -> None:
    client = create_client(api_key=os.environ["OPENAI_API_KEY"])
    request = ChatCompletionRequest.from_json(
        '{"model":"openai/gpt-4o","messages":[{"role":"user","content":"Tell me a story"}],"stream":true}'
    )
    async for chunk in client.chat_stream(request):
        if chunk.choices and chunk.choices[0].delta.content:
            print(chunk.choices[0].delta.content, end="", flush=True)
    print()

asyncio.run(main())

TypeScript

import { createClient } from "@xberg-io/liter-llm";

const client = createClient(process.env.OPENAI_API_KEY!);
const chunks = await client.chatStream({
  model: "openai/gpt-4o",
  messages: [{ role: "user", content: "Tell me a story" }],
});
for await (const chunk of chunks) {
  process.stdout.write(chunk.choices?.[0]?.delta?.content ?? "");
}

Notes

  • The first and last chunks often carry null content. Always null-check chunk.choices[0].delta.content (Python) or chunk.choices[0]?.delta?.content (TypeScript) before using it.
  • Tool-call deltas arrive in delta.tool_calls (Python) / delta.toolCalls (TypeScript); accumulate function.arguments fragments across chunks before parsing.
  • Through the proxy, request streaming with "stream": true on /v1/chat/completions.

Tool Calling

Pass a tools array of function definitions; the model may respond with tool_calls instead of (or alongside) text. Execute the named function and feed the result back as a tool message.

Python

import asyncio, json, os
from liter_llm import create_client
from liter_llm._internal_bindings import ChatCompletionRequest

payload = {
    "model": "openai/gpt-4o",
    "messages": [{"role": "user", "content": "What is the weather in Berlin?"}],
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Get the current weather for a location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {"type": "string", "description": "City name"},
                    },
                    "required": ["location"],
                },
            },
        }
    ],
    "tool_choice": "auto",
}

async def main() -> None:
    client = create_client(api_key=os.environ["OPENAI_API_KEY"])
    request = ChatCompletionRequest.from_json(json.dumps(payload))
    response = await client.chat(request)
    for call in response.choices[0].message.tool_calls or []:
        print(call.function.name, call.function.arguments)  # arguments is a JSON string

asyncio.run(main())

Structured outputs

Request strict JSON with response_format:

request = ChatCompletionRequest.from_json(json.dumps({
    "model": "openai/gpt-4o",
    "messages": [{"role": "user", "content": "Extract name and age as JSON."}],
    "response_format": {"type": "json_object"},
}))
response = await client.chat(request)

Notes

  • function.arguments is a JSON string — parse it before use.
  • Append each tool result as a message with role="tool" and the matching tool_call_id, then call chat again to let the model continue.
  • Tool support and JSON-mode availability vary by provider; check the provider reference if a model ignores tools.

Using the MCP Server

The liter-llm MCP server exposes 22 tools that mirror the proxy's REST endpoints, so an MCP-compatible client (Claude Code, Claude Desktop) can call 174 LLM providers as tools with no glue code.

Standalone skill installation

Installing this skill alone does not register an MCP server or include the plugin launcher. Install the CLI (for example, brew install xberg-io/tap/liter-llm), then configure your MCP client with liter-llm mcp --transport stdio as its stdio command. The full product plugin supplies registration where the runtime supports MCP.

How it runs in this plugin

The plugin auto-registers the server. Its command is the bundled launcher scripts/mcp-launch.sh, which execs liter-llm mcp --transport stdio. On first run the launcher resolves a liter-llm binary: it reuses one on PATH, then tries npx/uvx, then Homebrew, then downloads a prebuilt from the tool's latest GitHub release. Override with LITER_LLM_LAUNCHER=auto|npx|uvx|brew|download.

To run it manually:

liter-llm mcp --transport stdio          # for Claude Code / Claude Desktop
liter-llm mcp --transport http --port 3001

Configure model routing

Provider-backed tools use the proxy model registry. Create liter-llm-proxy.toml in the working directory, or pass --config /absolute/path/liter-llm-proxy.toml when launching:

[[models]]
name = "gpt-4o"
provider_model = "openai/gpt-4o"

Supply OPENAI_API_KEY in the MCP process environment, then call chat using the configured name gpt-4o. list_models shows configured names; an unknown model is a request error. Missing credentials are also reported on provider-backed requests. Gemini filters host credentials: install with --skip-settings and configure the selected provider through gemini extensions config liter-llm "OpenAI key". Local stdio does not need a separate proxy virtual key.

The 22 tools

Modality tools: chat, embed, generate_image, speech, transcribe, moderate, rerank, search, ocr.

Model listing: list_models.

File operations: create_file, list_files, retrieve_file, delete_file, file_content.

Batch operations: create_batch, list_batches, retrieve_batch, cancel_batch.

Responses API: create_response, retrieve_response, cancel_response.

Authentication

Local stdio trusts the invoking local client by default, so the registered command starts without extra proxy authentication configuration. Provider API keys still come from environment variables and are needed for provider-backed requests.

To apply a restricted virtual key, configure mcp.stdio_key_id; an unknown key fails startup. Set mcp.stdio_trust_local = false to require an explicit virtual key. HTTP transport retains its authentication policy; the local stdio default does not disable HTTP authorization.

When to prefer MCP over the CLI or SDK

  • Prefer MCP inside an agent session: the agent invokes tools directly, with no code to write or process to manage.
  • Prefer the SDK when building application software in a specific language.
  • Prefer the proxy when many apps or teams need a shared OpenAI-compatible endpoint with keys, budgets, and rate limits.

Liter-LLM Universal LLM Client

Liter-LLM is a universal LLM API client with a Rust core and native bindings for Python, TypeScript/Node.js, Go, Java, C#, Ruby, PHP, Elixir, WebAssembly, and C (FFI). One unified interface reaches 174 providers (OpenAI, Anthropic, Google Gemini, Groq, Mistral, Cohere, AWS Bedrock, Azure, and many more).

Capability map

  • Modalities — chat completions, streaming, tool/function calling, structured outputs, embeddings, image generation, audio (speech, transcription), moderation, web search, OCR, reranking, batch, and file operations.
  • Provider routing — provider/model prefix selects the backend (openai/gpt-4o, anthropic/claude-sonnet-4-20250514, gemini/gemini-2.0-flash). Set a full provider/model as model_hint to choose the client provider.
  • Middleware — response caching, rate limiting (RPM/TPM), cost tracking, budget enforcement (hard / soft), fallback chains, circuit-breaker cooldown, and background health checks.
  • Search providers (12) and OCR providers (4) — first-class search and ocr methods routed by the same prefix convention.
  • Proxy server — liter-llm api exposes an OpenAI-compatible gateway with 22 REST endpoints, virtual API keys, budgets, and SSE streaming.
  • MCP server — liter-llm mcp exposes 22 tools mirroring the proxy endpoints, for MCP-compatible clients (Claude Code, Claude Desktop).
  • 14 language bindings — same surface, language-native naming (snake_case for Python/Rust/Ruby/Go/Elixir/PHP, camelCase for TS/Node/WASM/C#/Java).

When to use the MCP server vs the SDK vs the proxy

Use the …When you want to …Entry point
MCP serverLet the agent call LLM APIs directly as tools, with no glue codeliter-llm mcp --transport stdio (this plugin auto-registers it)
SDK / bindingWrite application code that calls LLMs in a specific languagepip install liter-llm, cargo add liter-llm, etc.
ProxyGive many apps/teams a shared OpenAI-compatible endpoint with keys, budgets, and rate limitsliter-llm api --config liter-llm-proxy.toml

Rule of thumb: reach for the MCP server inside an agent session; the SDK when building software; the proxy when centralizing access for multiple consumers.

Installation

CLI (proxy + MCP server)

# Homebrew (macOS / Linux)
brew install xberg-io/tap/liter-llm

# or run it without a persistent install (the CLI proxy package self-installs the binary)
pnpm dlx @xberg-io/liter-llm-cli@2.2.3 --help

# or download a prebuilt binary from the latest GitHub release:
#   https://github.com/xberg-io/liter-llm/releases/latest

# or build from source
cargo install liter-llm-cli

# or Docker (35MB image)
docker pull ghcr.io/xberg-io/liter-llm

Language bindings

LanguageInstall
Pythonpip install liter-llm
Node.jspnpm add @xberg-io/liter-llm
Rustcargo add liter-llm
Gogo get github.com/xberg-io/liter-llm/packages/go/v2
Rubygem install liter_llm
PHPcomposer require xberg-io/liter-llm
C#dotnet add package XbergIo.LiterLlm
WASMpnpm add @xberg-io/liter-llm-wasm

Quick start (Python, async)

import asyncio
import os
from liter_llm import create_client
from liter_llm._internal_bindings import ChatCompletionRequest

async def main() -> None:
    client = create_client(api_key=os.environ["OPENAI_API_KEY"])
    request = ChatCompletionRequest.from_json(
        '{"model":"openai/gpt-4o","messages":[{"role":"user","content":"Hello!"}]}'
    )
    response = await client.chat(request)
    print(response.choices[0].message.content)

asyncio.run(main())

Provider routing

The prefix before / in the request's model selects the provider:

ChatCompletionRequest.from_json('{"model":"openai/gpt-4o","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"anthropic/claude-sonnet-4-20250514","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"gemini/gemini-2.0-flash","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"groq/llama3-70b","messages":[...]}')

Client construction resolves credentials for its selected provider from environment variables: OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, GROQ_API_KEY, MISTRAL_API_KEY, COHERE_API_KEY, and AWS credentials for Bedrock. Create a separate client for each provider with matching model_hint and credentials. Request prefixes change routing but reuse the client API key; they do not select a new environment key per request.

Configuration

Pass basic options to create_client(...) (api_key, base_url, timeout_secs, connect_timeout_secs, max_retries, model_hint), or set the full surface in a liter-llm.toml (SDK) / liter-llm-proxy.toml (proxy) — both auto-discover from the cwd upward.

# liter-llm.toml
api_key = "${OPENAI_API_KEY}"
model_hint = "openai/gpt-4o"
timeout_secs = 120
max_retries = 5

[cache]
max_entries = 512
ttl_seconds = 600

[budget]
global_limit = 50.0
enforcement = "hard"

[rate_limit]
rpm = 60
tpm = 100000
OptionDefaultDescription
api_keyrequiredProvider key, wrapped in SecretString (never logged).
base_urlfrom registryOverride the provider base URL.
model_hintnoneFull provider/model used to select the client provider and credentials.
timeout_secs60Request timeout in seconds.
connect_timeout_secs10Connect timeout in seconds. A connection that does not open in time is retried up to max_retries times. Zero is an error on every target. wasm32 ignores any other value.
max_retries3Retries on 429/5xx with exponential backoff.
cachenonemax_entries, ttl_seconds.
budgetnoneglobal_limit, model_limits, enforcement.
rate_limitnonerpm, tpm.
cost_trackingfalsePer-request cost tracking.
tracingfalseOpenTelemetry spans.

Proxy server

liter-llm api --config liter-llm-proxy.toml

22 OpenAI-compatible endpoints, model routing by prefix, virtual API keys, per-key RPM/TPM limits, cost tracking, budget enforcement, response caching, SSE streaming, and an OpenAPI 3.1 spec at /openapi.json.

MCP server

liter-llm mcp --transport stdio   # for Claude Code / Claude Desktop
liter-llm mcp --transport http --port 3001

This plugin auto-registers the stdio server via scripts/mcp-launch.sh. The 22 tools mirror the proxy endpoints: chat, embed, generate_image, speech, transcribe, moderate, rerank, search, ocr, list_models; file ops (create_file, list_files, retrieve_file, delete_file, file_content); batch ops (create_batch, list_batches, retrieve_batch, cancel_batch); and Responses API (create_response, retrieve_response, cancel_response).

Common pitfalls

  1. Provider routing — use "provider/model" to select a request backend. Bare model names fall back to the configured client provider (OpenAI by default).
  2. API keys are SecretString — never logged or serialized. Read from env vars; never hardcode.
  3. Python methods are async — await inside an async context.
  4. Naming conventions differ — camelCase for TS/Node/WASM/C#/Java, snake_case elsewhere.
  5. Streaming chunks may have null content — null-check chunk.choices[0].delta.content before use.
  6. Budget modes — hard rejects over-budget requests; soft logs and allows.

Additional resources

Commands 2

Help with this chat task: $ARGUMENTS

Read the calling-llms skill. Choose a provider/model, use the supported SDK or proxy request shape, and read API keys from environment variables. Never print keys.

Implement this streaming task: $ARGUMENTS

Read the streaming-responses skill. Use the selected language binding or proxy streaming API, handle incremental responses and errors, and avoid buffering the whole result.

MCP servers 1

Sponsors

Details

Marketplace
xberg-io/liter-llm
Source
plugin
Category
Community