API reference

Chat completions and embeddings follow the OpenAI schema. Rerank follows the Cohere schema. View live endpoints in the Live catalog.

Conventions#

Every response carries an X-Litellm-Call-Id header. It identifies a request without reference to its content, and is present on errors as well as successes. Quote it in support mail.

Errors#

Errors return the OpenAI error object on every route, including unmatched paths:

{"error": {"message": "...", "type": "...", "param": "...", "code": "..."}}

Branch on the HTTP status. message is human-readable; type, param and code are informational and code carries the status as a string.

An unrecognised request field is ignored, not refused. A misspelled max_token is dropped and the request runs with the default, so check field names against this reference.

Rate limits#

Limits are set per API key when the key is issued: a ceiling on simultaneous requests plus a requests-per-minute and tokens-per-minute budget. Every response carries the current state:

Budgets are per calendar minute and refill at the top of the next one. Requests beyond a limit are shed with 429 and Retry-After: 2, never queued. In-flight requests are unaffected.

A concurrency ceiling is a ceiling, not a rate to sit at. A client running exactly that many requests in a loop transiently presents one more than the pool holds. Sit one below the published number, or handle the occasional 429.

A key's rate budget is shared across every model it can reach, so a caller driving several models at once is best served by one key per model.

POST /v1/chat/completions#

Content safety. Standard OpenAI chat completions, streaming and non-streaming. With stream_options: {"include_usage": true} the final chunk before [DONE] carries exact usage.

The guard model judges the conversation against one criterion at a time, selected through chat_template_kwargs:

response = client.chat.completions.create(
    model="granite-guardian-3.3-8b",
    messages=[{"role": "user", "content": "..."}],
    extra_body={"chat_template_kwargs": {
        "guardian_config": {"criteria_id": "harm"},
        "think": False,
    }},
)
# -> "<think>\n</think>\n<score> yes </score>"

Response content is <score>yes</score> or <score>no</score>, preceded by a <think>...</think> block when think is true.

criteria_id must be one of these 13. Anything else returns 400:

answer_relevance, context_relevance, evasiveness, function_call, groundedness, harm, harm_engagement, jailbreak, profanity, sexual_content, social_bias, unethical_behavior, violence

Omitting guardian_config judges against harm. Criteria that judge a response rather than a prompt (groundedness, answer_relevance, evasiveness, harm_engagement, function_call) score the last assistant message. Pass retrieved passages as chat_template_kwargs.documents and tool definitions as chat_template_kwargs.available_tools.

Limits#

Response#

{
  "id": "chatcmpl-8a932b0e1ed1aa0f",
  "object": "chat.completion",
  "created": 1789999999,
  "model": "granite-guardian-3.3-8b",
  "choices": [
    {
      "index": 0,
      "message": {"role": "assistant", "content": "<think>\n</think>\n<score> yes </score>"},
      "finish_reason": "stop"
    }
  ],
  "usage": {"prompt_tokens": 412, "completion_tokens": 12, "total_tokens": 424}
}

POST /v1/embeddings#

Retrieval. OpenAI schema. Fixed-width, L2-normalised vectors of 2,560 dimensions on every call.

client.embeddings.create(model="qwen3-embedding-4b", input=["...", "..."])

Limits: 8,192 tokens per input. Over it returns 400. usage.prompt_tokens comes from the engine.

For retrieval quality matching the model's intended use, prefix queries with Instruct: <task>\nQuery: and leave documents unprefixed.

POST /v1/rerank#

Retrieval. Cohere convention.

requests.post(f"{BASE}/rerank", headers=auth, json={
    "model": "qwen3-reranker-0.6b",
    "query": "What sets winter electricity prices?",
    "documents": ["...", "...", "..."],
    "top_n": 3,
})

{"id": "...", "model": "qwen3-reranker-0.6b",
 "results": [{"index": 2, "relevance_score": 0.971}, ...],
 "usage": {"prompt_tokens": 11, "total_tokens": 11}}

Results are ordered by descending relevance_score. The score is a calibrated probability that the document answers the query. Use it for ordering, not as a fixed-threshold filter: a well-formed question scores a relevant document around 0.97, while a bare keyword fragment scores the same document near zero.

document is omitted by default. Set return_documents: true to have documents echoed.

Limits: 8,192 tokens per query-and-document pair.

GET /v1/models#

Keyed. Returns the models currently served in the stock OpenAI shape: one entry per model, carrying its id.

{"object": "list", "data": [
  {"id": "granite-guardian-3.3-8b", "object": "model"},
  {"id": "qwen3-embedding-4b", "object": "model"},
  {"id": "qwen3-reranker-0.6b", "object": "model"}
]}

GET /v1/catalog.json#

Public, no key, cached five minutes. One document per deploy, joining what the product declares with what each model actually loaded. This is what the Live catalog reads.

{"object": "list", "generated_at": "2026-09-18T22:37:10+00:00", "data": [{
  "id": "qwen3-embedding-4b",
  "route": "embeddings",
  "category": "retrieval",
  "datacenter": "SE",
  "data_policy": "zero-retention",
  "upstream": "Qwen/Qwen3-Embedding-4B",
  "revision": "5cf2132abc99cad020ac570b19d031efec650f2b",
  "context_length": 8192,
  "embedding_dimensions": 2560,
  "quantization": "compressed-tensors per-tensor",
  "limits": {"max_concurrency_per_key": 12, "max_requests_per_minute": 250},
  "pricing": {"input_usd_per_1m": null, "output_usd_per_1m": null},
  "declared_rpm": 190, "declared_tpm": 97000, "declared_latency_p95_s": 0.5
}]}

Health#