The ManyLayers AI Gateway is a single OpenAI-compatible endpoint that sits between your applications and the model providers you use. You point any OpenAI SDK at it, and it authenticates the caller, applies your access rules, limits, guardrails and PII redaction, routes the request to the right provider, and records cost, metrics and a tamper-evident audit trail.

​
Key features

Making requests

Use the OpenAI SDK unchanged: chat, completions, embeddings, responses, streaming.

Anthropic Messages API

Call /v1/messages in Anthropic format and serve it from any provider.

Providers

Connect providers from OpenAI and Bedrock to vLLM and Ollama.

Virtual models

Expose a routing config under a model name your clients can call directly.

Routing and fallback

Fallback, load balancing, canary, conditional and latency-based routing.

Auto routing

Classify each turn’s complexity and send it to the right tier of model.

Caching

Exact and semantic response caching, keyed on the redacted request.

Async requests

Queue completions with /v1/async/*, poll for the result or get a webhook.

Access control

Roles, team model allow-lists and provider account access.

Rate limits

Request and token limits per minute, hour or day at every level.

Budgets

Spending caps in USD or tokens; the most restrictive one wins.

Guardrails

Validate and mutate inputs and outputs with built-in or custom checks.

PII redaction

Redact personal data before it reaches a provider, with an optional reversible vault.

Prompt injection

Score prompts for injection attempts and audit or block them.

Tracing

Send OpenTelemetry traces to your own collector per workspace.

Metrics

Prometheus metrics for requests, tokens, cache, policy and upstream health.

Self-hosting

Run the gateway, workspace and deployer services on your own infrastructure.

​
Supported providers

Each provider type has an adapter or an OpenAI-wire passthrough. Passthrough providers receive the request body unchanged, so the provider’s own API decides what it accepts. Register them in the console under Providers, or declare them in gateway.yaml when you self-host.
Familyprovider_type values
OpenAI and Azure OpenAIopenai, azure
Anthropicanthropic, aws_claude_platform
Googlegemini, vertex
Amazonbedrock, bedrock_mantle, sagemaker
Data platformsdatabricks, snowflake_cortex, cloudera
Other native adapterscohere, typesafe
Hosted open-model inference (OpenAI wire)mistral, groq, deepseek, together, perplexity, xai, fireworks, openrouter, deepinfra, baseten, cerebras, ai21, sambanova, wafer
Your own servers (vLLM, TGI, LM Studio, Ollama, …)self_hosted, ollama
Speech onlyelevenlabs, deepgram, cartesia, smallest
See Providers for what each type can serve, its credential format and its default endpoint.
For adapters that translate to a native API (such as Anthropic, Gemini, Vertex, Bedrock, Cohere and TypeSafe), the gateway checks each request parameter it cannot translate faithfully, such as n, seed or response_format, and returns 400 unsupported_parameter naming that parameter. It never drops the parameter silently. TypeSafe requires response_format with a JSON schema.
Providers you register in the console serve /v1/chat/completions, /v1/responses, /v1/messages, /v1/embeddings, /v1/rerank, /v1/audio/speech and /v1/audio/transcriptions for keys bound to that workspace. Embeddings, rerank and speech fall back to models declared under models: in gateway.yaml when the workspace has no model of that name. A workspace-bound key never resolves chat models from gateway.yaml. /v1/completions, images, /v1/audio/translations, moderations, files, fine-tuning and realtime resolve only from gateway.yaml models, so they are available only on a self-hosted gateway that declares such models.

​
Supported APIs

On ManyLayers Cloud every endpoint lives under https://app.manylayers.io/v1 and accepts Authorization: Bearer <key>. A self-hosted gateway serves the same paths on your own host.
EndpointMethodNotes
/v1/chat/completionsPOSTOpenAI Chat Completions, streaming supported
/v1/completionsPOSTLegacy text completions
/v1/embeddingsPOSTEmbeddings
/v1/responsesPOSTOpenAI Responses API, stateless; translated to chat so it works with every chat provider
/v1/messagesPOSTAnthropic Messages format, including SSE, served by any chat provider
/v1/models, /v1/models/{model}GETModels the caller can use
/v1/async/chat/completions, /v1/async/completionsPOSTEnqueue a job
/v1/async/jobs/{id}GETPoll a queued job
/v1/realtimeGET (WebSocket)Realtime API, openai and azure only; pass ?model=
/v1/images/generations, /edits, /variationsPOSTopenai and azure only
/v1/audio/speech, /transcriptionsPOSTopenai, azure, elevenlabs, deepgram, cartesia, smallest
/v1/audio/translationsPOSTopenai and azure only
/v1/moderationsPOSTPassthrough to openai/azure, built-in fallback otherwise
/v1/rerankPOSTTranslated for Cohere, passthrough for OpenAI-wire providers
/v1/files, /v1/fine_tuning/jobsGET, POST, DELETEopenai and azure only; pick the model with X-ManyLayers-Model or ?model=
/v1/systemonePOSTTypeSafe’s native System One call
/v1/kb, /v1/docsetsGET, POST, PUT, DELETEKnowledge bases and document sets
The stateful OpenAI surfaces (/v1/assistants, /v1/threads, /v1/vector_stores, /v1/conversations, /v1/containers, /v1/uploads, /v1/evals, /v1/organization) are refused with 404 unsupported_endpoint in the standard OpenAI error envelope, and so is any other unrouted path under /v1. The gateway also serves /health, /healthz, /ready, /readyz, /version and /metrics.

​
Next steps

Quick Start

Add a provider, create a key and send your first request.

Authentication

API keys, personal access tokens, virtual accounts and OIDC.

Providers

Configure each provider type in detail.