
liter-llm
Universal LLM API client for 174 providers — chat, streaming, tools, embeddings, search, OCR, plus an OpenAI-compatible proxy and an MCP server.
- GitHub stars
- 266
Install
/plugin install liter-llm@liter-llmRun it inside Claude Code after adding the marketplace with /plugin marketplace add xberg-io/liter-llm.
Contents
Discovered from sourceSkills 7
Calling LLMs
Build a ChatCompletionRequest and send it with client.chat(request). Create
the client with create_client(...). The model string is provider/model; the
prefix selects the backend.
import asyncio, json, os
from liter_llm import create_client
from liter_llm._internal_bindings import ChatCompletionRequest
async def main() -> None:
client = create_client(api_key=os.environ["OPENAI_API_KEY"])
request = ChatCompletionRequest.from_json(json.dumps({
"model": "openai/gpt-4o",
"messages": [
{"role": "system", "content": "You are concise."},
{"role": "user", "content": "Name three Rust crates for HTTP."},
],
}))
response = await client.chat(request)
print(response.choices[0].message.content)
asyncio.run(main())
Provider routing
The model string's prefix selects the provider; build a request per backend:
ChatCompletionRequest.from_json('{"model":"anthropic/claude-sonnet-4-20250514","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"gemini/gemini-2.0-flash","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"groq/llama3-70b","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"mistral/mistral-large-latest","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"bedrock/anthropic.claude-v2","messages":[...]}')
Set a full provider/model as model_hint at construction to select the client
provider and credential source. Requests without a recognized prefix fall back to
that client provider:
client = create_client(api_key=os.environ["OPENAI_API_KEY"], model_hint="openai/gpt-4o")
# the request model can now omit the provider prefix:
request = ChatCompletionRequest.from_json('{"model":"gpt-4o","messages":[...]}')
await client.chat(request) # routes to OpenAI
Notes
- Keys come from env vars (
OPENAI_API_KEY,ANTHROPIC_API_KEY, …); never hardcode them. - Without a recognized prefix, requests fall back to the client's configured provider;
a client without
model_hintdefaults to OpenAI. - Environment credentials are selected when constructing the client, not independently
for every request. Use a separate client with matching
model_hintand credentials for each provider. A prefix changes routing but does not switch the client's API key. - Python errors are typed exceptions exported from
liter_llm:AuthenticationError,RateLimitedError,BadRequestError,ContextWindowExceededError,ContentPolicyError,NotFoundError,ServerError,ServiceUnavailableError,LiterLlmTimeoutError,BudgetExceededError— all subclasses ofLiterLlmError.
Embeddings and Search
liter-llm exposes embeddings, web search (12 providers), OCR (4 providers), and
reranking through the same provider/model routing convention.
Embeddings
import asyncio, os
from liter_llm import create_client
from liter_llm._internal_bindings import EmbeddingRequest
async def main() -> None:
client = create_client(api_key=os.environ["OPENAI_API_KEY"])
request = EmbeddingRequest.from_json(
'{"model":"openai/text-embedding-3-small","input":["first document","second document"]}'
)
response = await client.embed(request)
for item in response.data:
print(len(item.embedding))
asyncio.run(main())
Many embedding models support dimension selection and base64 output; set
dimensions / encoding_format in the request where the provider allows it.
Web search (12 providers)
from liter_llm._internal_bindings import SearchRequest
client = create_client(api_key=os.environ["BRAVE_API_KEY"])
request = SearchRequest.from_json(
'{"model":"brave/web-search","query":"What is the Rust programming language?","max_results":5}'
)
response = await client.search(request)
for result in response.results:
print(result.title, result.url)
OCR (4 providers)
from liter_llm._internal_bindings import OcrRequest
client = create_client(api_key=os.environ["MISTRAL_API_KEY"])
request = OcrRequest.from_json(
'{"model":"mistral/mistral-ocr-latest",'
'"document":{"type":"document_url","url":"https://example.com/invoice.pdf"}}'
)
response = await client.ocr(request)
for page in response.pages:
print(page.index, page.markdown[:100])
Reranking
Build a RerankRequest (model, query, documents) and call client.rerank(request)
to score and order candidate documents against a query for retrieval pipelines —
combine it with embed for hybrid retrieval. Each result carries index and
relevance_score. Routing follows the same provider/model convention.
Notes
- Search and OCR providers each need their own API key (e.g.
BRAVE_API_KEY,MISTRAL_API_KEY); read them from env vars. - See the upstream provider reference for the full list of the 12 search and 4 OCR backends and their model identifiers.
Running the Proxy
liter-llm api is a drop-in OpenAI-compatible gateway: 22 REST endpoints that
route to 174 providers, with multi-tenant virtual keys, rate limits, budgets,
and cost tracking.
Start it
liter-llm api --config liter-llm-proxy.toml
The proxy auto-discovers liter-llm-proxy.toml in the current directory.
Configuration
[server]
host = "0.0.0.0"
port = 4000
[general]
master_key = "${LITER_LLM_MASTER_KEY}"
# Each [[models]] entry maps a routable name to a provider/model and its key.
[[models]]
name = "gpt-4o"
provider_model = "openai/gpt-4o"
api_key = "${OPENAI_API_KEY}"
[[models]]
name = "claude-sonnet"
provider_model = "anthropic/claude-sonnet-4-20250514"
api_key = "${ANTHROPIC_API_KEY}"
# Virtual keys scope which configured model names a caller may use.
[[keys]]
key = "sk-team-frontend"
models = ["gpt-4o", "claude-sonnet"]
rpm = 60
tpm = 100000
budget_limit = 50.0
${ENV_VAR} interpolation keeps secrets out of the file.
Call it like OpenAI
curl http://localhost:4000/v1/chat/completions \
-H "Authorization: Bearer sk-team-frontend" \
-d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "Hello"}]}'
Notes
- Virtual keys scope which models a caller may use and carry their own RPM/TPM
and budget limits; the
master_keyadministers them. - The OpenAPI 3.1 spec is served at
/openapi.json. - Endpoints cover chat, embeddings, images, audio, moderations, files, batches, responses, and model listing.
- Docker:
docker run -p 4000:4000 -e LITER_LLM_MASTER_KEY=sk-key ghcr.io/xberg-io/liter-llm.
Streaming Responses
Use chat_stream(...) to receive tokens as they are produced instead of waiting
for the full completion. The proxy streams over SSE; bindings expose async
iterators.
Python
import asyncio, os
from liter_llm import create_client
from liter_llm._internal_bindings import ChatCompletionRequest
async def main() -> None:
client = create_client(api_key=os.environ["OPENAI_API_KEY"])
request = ChatCompletionRequest.from_json(
'{"model":"openai/gpt-4o","messages":[{"role":"user","content":"Tell me a story"}],"stream":true}'
)
async for chunk in client.chat_stream(request):
if chunk.choices and chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
print()
asyncio.run(main())
TypeScript
import { createClient } from "@xberg-io/liter-llm";
const client = createClient(process.env.OPENAI_API_KEY!);
const chunks = await client.chatStream({
model: "openai/gpt-4o",
messages: [{ role: "user", content: "Tell me a story" }],
});
for await (const chunk of chunks) {
process.stdout.write(chunk.choices?.[0]?.delta?.content ?? "");
}
Notes
- The first and last chunks often carry null content. Always null-check
chunk.choices[0].delta.content(Python) orchunk.choices[0]?.delta?.content(TypeScript) before using it. - Tool-call deltas arrive in
delta.tool_calls(Python) /delta.toolCalls(TypeScript); accumulatefunction.argumentsfragments across chunks before parsing. - Through the proxy, request streaming with
"stream": trueon/v1/chat/completions.
Tool Calling
Pass a tools array of function definitions; the model may respond with
tool_calls instead of (or alongside) text. Execute the named function and feed
the result back as a tool message.
Python
import asyncio, json, os
from liter_llm import create_client
from liter_llm._internal_bindings import ChatCompletionRequest
payload = {
"model": "openai/gpt-4o",
"messages": [{"role": "user", "content": "What is the weather in Berlin?"}],
"tools": [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {"type": "string", "description": "City name"},
},
"required": ["location"],
},
},
}
],
"tool_choice": "auto",
}
async def main() -> None:
client = create_client(api_key=os.environ["OPENAI_API_KEY"])
request = ChatCompletionRequest.from_json(json.dumps(payload))
response = await client.chat(request)
for call in response.choices[0].message.tool_calls or []:
print(call.function.name, call.function.arguments) # arguments is a JSON string
asyncio.run(main())
Structured outputs
Request strict JSON with response_format:
request = ChatCompletionRequest.from_json(json.dumps({
"model": "openai/gpt-4o",
"messages": [{"role": "user", "content": "Extract name and age as JSON."}],
"response_format": {"type": "json_object"},
}))
response = await client.chat(request)
Notes
function.argumentsis a JSON string — parse it before use.- Append each tool result as a message with
role="tool"and the matchingtool_call_id, then callchatagain to let the model continue. - Tool support and JSON-mode availability vary by provider; check the provider
reference if a model ignores
tools.
Using the MCP Server
The liter-llm MCP server exposes 22 tools that mirror the proxy's REST
endpoints, so an MCP-compatible client (Claude Code, Claude Desktop) can call
174 LLM providers as tools with no glue code.
Standalone skill installation
Installing this skill alone does not register an MCP server or include the plugin launcher.
Install the CLI (for example, brew install xberg-io/tap/liter-llm), then configure your MCP client with
liter-llm mcp --transport stdio as its stdio command. The full product plugin supplies registration where
the runtime supports MCP.
How it runs in this plugin
The plugin auto-registers the server. Its command is the bundled launcher
scripts/mcp-launch.sh, which execs liter-llm mcp --transport stdio. On first
run the launcher resolves a liter-llm binary: it reuses one on PATH, then
tries npx/uvx, then Homebrew, then downloads a prebuilt
from the tool's latest GitHub release. Override with
LITER_LLM_LAUNCHER=auto|npx|uvx|brew|download.
To run it manually:
liter-llm mcp --transport stdio # for Claude Code / Claude Desktop
liter-llm mcp --transport http --port 3001
Configure model routing
Provider-backed tools use the proxy model registry. Create liter-llm-proxy.toml in the
working directory, or pass --config /absolute/path/liter-llm-proxy.toml when launching:
[[models]]
name = "gpt-4o"
provider_model = "openai/gpt-4o"
Supply OPENAI_API_KEY in the MCP process environment, then call chat using the
configured name gpt-4o. list_models shows configured names; an unknown model is a
request error. Missing credentials are also reported on provider-backed requests.
Gemini filters host credentials: install with --skip-settings and configure the selected
provider through gemini extensions config liter-llm "OpenAI key". Local stdio does not
need a separate proxy virtual key.
The 22 tools
Modality tools: chat, embed, generate_image, speech, transcribe,
moderate, rerank, search, ocr.
Model listing: list_models.
File operations: create_file, list_files, retrieve_file, delete_file,
file_content.
Batch operations: create_batch, list_batches, retrieve_batch,
cancel_batch.
Responses API: create_response, retrieve_response, cancel_response.
Authentication
Local stdio trusts the invoking local client by default, so the registered command starts without extra proxy authentication configuration. Provider API keys still come from environment variables and are needed for provider-backed requests.
To apply a restricted virtual key, configure mcp.stdio_key_id; an unknown key fails startup.
Set mcp.stdio_trust_local = false to require an explicit virtual key. HTTP transport retains
its authentication policy; the local stdio default does not disable HTTP authorization.
When to prefer MCP over the CLI or SDK
- Prefer MCP inside an agent session: the agent invokes tools directly, with no code to write or process to manage.
- Prefer the SDK when building application software in a specific language.
- Prefer the proxy when many apps or teams need a shared OpenAI-compatible endpoint with keys, budgets, and rate limits.
Liter-LLM Universal LLM Client
Liter-LLM is a universal LLM API client with a Rust core and native bindings for Python, TypeScript/Node.js, Go, Java, C#, Ruby, PHP, Elixir, WebAssembly, and C (FFI). One unified interface reaches 174 providers (OpenAI, Anthropic, Google Gemini, Groq, Mistral, Cohere, AWS Bedrock, Azure, and many more).
Capability map
- Modalities — chat completions, streaming, tool/function calling, structured outputs, embeddings, image generation, audio (speech, transcription), moderation, web search, OCR, reranking, batch, and file operations.
- Provider routing —
provider/modelprefix selects the backend (openai/gpt-4o,anthropic/claude-sonnet-4-20250514,gemini/gemini-2.0-flash). Set a fullprovider/modelasmodel_hintto choose the client provider. - Middleware — response caching, rate limiting (RPM/TPM), cost tracking,
budget enforcement (
hard/soft), fallback chains, circuit-breaker cooldown, and background health checks. - Search providers (12) and OCR providers (4) — first-class
searchandocrmethods routed by the same prefix convention. - Proxy server —
liter-llm apiexposes an OpenAI-compatible gateway with 22 REST endpoints, virtual API keys, budgets, and SSE streaming. - MCP server —
liter-llm mcpexposes 22 tools mirroring the proxy endpoints, for MCP-compatible clients (Claude Code, Claude Desktop). - 14 language bindings — same surface, language-native naming (snake_case for Python/Rust/Ruby/Go/Elixir/PHP, camelCase for TS/Node/WASM/C#/Java).
When to use the MCP server vs the SDK vs the proxy
| Use the … | When you want to … | Entry point |
|---|---|---|
| MCP server | Let the agent call LLM APIs directly as tools, with no glue code | liter-llm mcp --transport stdio (this plugin auto-registers it) |
| SDK / binding | Write application code that calls LLMs in a specific language | pip install liter-llm, cargo add liter-llm, etc. |
| Proxy | Give many apps/teams a shared OpenAI-compatible endpoint with keys, budgets, and rate limits | liter-llm api --config liter-llm-proxy.toml |
Rule of thumb: reach for the MCP server inside an agent session; the SDK when building software; the proxy when centralizing access for multiple consumers.
Installation
CLI (proxy + MCP server)
# Homebrew (macOS / Linux)
brew install xberg-io/tap/liter-llm
# or run it without a persistent install (the CLI proxy package self-installs the binary)
pnpm dlx @xberg-io/liter-llm-cli@2.2.3 --help
# or download a prebuilt binary from the latest GitHub release:
# https://github.com/xberg-io/liter-llm/releases/latest
# or build from source
cargo install liter-llm-cli
# or Docker (35MB image)
docker pull ghcr.io/xberg-io/liter-llm
Language bindings
| Language | Install |
|---|---|
| Python | pip install liter-llm |
| Node.js | pnpm add @xberg-io/liter-llm |
| Rust | cargo add liter-llm |
| Go | go get github.com/xberg-io/liter-llm/packages/go/v2 |
| Ruby | gem install liter_llm |
| PHP | composer require xberg-io/liter-llm |
| C# | dotnet add package XbergIo.LiterLlm |
| WASM | pnpm add @xberg-io/liter-llm-wasm |
Quick start (Python, async)
import asyncio
import os
from liter_llm import create_client
from liter_llm._internal_bindings import ChatCompletionRequest
async def main() -> None:
client = create_client(api_key=os.environ["OPENAI_API_KEY"])
request = ChatCompletionRequest.from_json(
'{"model":"openai/gpt-4o","messages":[{"role":"user","content":"Hello!"}]}'
)
response = await client.chat(request)
print(response.choices[0].message.content)
asyncio.run(main())
Provider routing
The prefix before / in the request's model selects the provider:
ChatCompletionRequest.from_json('{"model":"openai/gpt-4o","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"anthropic/claude-sonnet-4-20250514","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"gemini/gemini-2.0-flash","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"groq/llama3-70b","messages":[...]}')
Client construction resolves credentials for its selected provider from environment variables: OPENAI_API_KEY, ANTHROPIC_API_KEY,
GEMINI_API_KEY, GROQ_API_KEY, MISTRAL_API_KEY, COHERE_API_KEY, and AWS
credentials for Bedrock. Create a separate client for each provider with matching
model_hint and credentials. Request prefixes change routing but reuse the client
API key; they do not select a new environment key per request.
Configuration
Pass basic options to create_client(...) (api_key, base_url,
timeout_secs, connect_timeout_secs, max_retries, model_hint), or set the full surface in a
liter-llm.toml (SDK) / liter-llm-proxy.toml (proxy) — both auto-discover
from the cwd upward.
# liter-llm.toml
api_key = "${OPENAI_API_KEY}"
model_hint = "openai/gpt-4o"
timeout_secs = 120
max_retries = 5
[cache]
max_entries = 512
ttl_seconds = 600
[budget]
global_limit = 50.0
enforcement = "hard"
[rate_limit]
rpm = 60
tpm = 100000
| Option | Default | Description |
|---|---|---|
api_key | required | Provider key, wrapped in SecretString (never logged). |
base_url | from registry | Override the provider base URL. |
model_hint | none | Full provider/model used to select the client provider and credentials. |
timeout_secs | 60 | Request timeout in seconds. |
connect_timeout_secs | 10 | Connect timeout in seconds. A connection that does not open in time is retried up to max_retries times. Zero is an error on every target. wasm32 ignores any other value. |
max_retries | 3 | Retries on 429/5xx with exponential backoff. |
cache | none | max_entries, ttl_seconds. |
budget | none | global_limit, model_limits, enforcement. |
rate_limit | none | rpm, tpm. |
cost_tracking | false | Per-request cost tracking. |
tracing | false | OpenTelemetry spans. |
Proxy server
liter-llm api --config liter-llm-proxy.toml
22 OpenAI-compatible endpoints, model routing by prefix, virtual API keys,
per-key RPM/TPM limits, cost tracking, budget enforcement, response caching, SSE
streaming, and an OpenAPI 3.1 spec at /openapi.json.
MCP server
liter-llm mcp --transport stdio # for Claude Code / Claude Desktop
liter-llm mcp --transport http --port 3001
This plugin auto-registers the stdio server via scripts/mcp-launch.sh. The 22
tools mirror the proxy endpoints: chat, embed, generate_image, speech,
transcribe, moderate, rerank, search, ocr, list_models; file ops
(create_file, list_files, retrieve_file, delete_file, file_content);
batch ops (create_batch, list_batches, retrieve_batch, cancel_batch);
and Responses API (create_response, retrieve_response, cancel_response).
Common pitfalls
- Provider routing — use
"provider/model"to select a request backend. Bare model names fall back to the configured client provider (OpenAI by default). - API keys are SecretString — never logged or serialized. Read from env vars; never hardcode.
- Python methods are async —
awaitinside an async context. - Naming conventions differ — camelCase for TS/Node/WASM/C#/Java, snake_case elsewhere.
- Streaming chunks may have null content — null-check
chunk.choices[0].delta.contentbefore use. - Budget modes —
hardrejects over-budget requests;softlogs and allows.
Additional resources
- Upstream docs: https://docs.liter-llm.xberg.io
- GitHub: https://github.com/xberg-io/liter-llm
Commands 2
Help with this chat task: $ARGUMENTS
Read the calling-llms skill. Choose a provider/model, use the supported SDK or proxy request shape, and read API keys from environment variables. Never print keys.
Implement this streaming task: $ARGUMENTS
Read the streaming-responses skill. Use the selected language binding or proxy streaming API, handle incremental responses and errors, and avoid buffering the whole result.
MCP servers 1
Sponsors
Details
- Marketplace
- xberg-io/liter-llm
- Source
- plugin
- Category
- Community