What is an LLM Gateway?

A proxy between an application and its model providers. It handles routing, keys, logging, rate limits, and pattern-based redaction. A gateway can mask a phone number. It cannot tell whether a sentence violates the Marketing Rule.

How it works

Four functions. One connection to your models.

Routing

Directs requests to the appropriate model or provider.

Access

Manages API keys and controls access to model providers.

Rate Limiting

Applies configured limits to incoming requests.

Logging

Records request activity for monitoring and troubleshooting.

Requests pass through the gateway on their way to a model provider, and responses return through the same connection.

What it is not

Adjacent tools, and where they stop.

Routing is not rule evaluation.

Directing a request to a model does not establish whether its response follows written rules.

Redaction matches patterns.

Masking sensitive information is different from evaluating a message against a regulation or company policy.

Rate limits control usage.

Limiting requests does not determine whether the content of a response is compliant.

Logging records activity.

A record of a message does not, by itself, prevent a violation before delivery.

Why it matters

More AI applications. More infrastructure to manage.

14

GenAI use-case categories identified by FINRA among its member companies, including conversational AI, content generation, and workflow automation.

Source: FINRA, How FINRA Member Firms Use GenAI (July 2025)

24-09

FINRA Regulatory Notice reminding member companies that existing regulatory obligations continue to apply when they use generative AI and large language models.

Source: FINRA, Regulatory Notice 24-09 (June 2024)

As companies use language models for more tasks, managing connections to different model providers becomes increasingly important. An LLM gateway gives developers a central place to route requests, manage access, and monitor usage. But managing model traffic does not, by itself, establish whether an AI-generated message meets regulatory requirements or company policies. That requires evaluating the content against the applicable rules.

FAQ

Common questions.

What is the purpose of an LLM gateway?

An LLM gateway provides a central connection between an AI application and its model providers. It helps developers manage requests, API keys, rate limits, and usage across different models without maintaining separate integrations for each provider.

How does an LLM gateway work?

An application sends a request to the gateway, which routes it to a model provider according to its configuration. The provider processes the request, and the response returns through the gateway. Depending on the implementation, the gateway may also log activity, cache responses, or filter content.

Can an LLM gateway connect to multiple model providers?

Yes. Many LLM gateways support multiple providers through a common interface. Developers can configure routing between supported models, although the available providers and routing options vary by gateway.

Is an LLM gateway the same as AI guardrails?

No. An LLM gateway primarily manages connections and traffic between applications and model providers. AI guardrails check inputs or outputs for unsafe or unwanted content. Some gateways include guardrail features, so their capabilities can overlap.

Can an LLM gateway enforce company policies?

Some gateways support content filtering and policy checks. However, routing requests, managing API keys, and applying rate limits do not establish whether a generated message follows written rules. Evaluating compliance requires checking the message against the applicable requirements.

How is an LLM gateway different from an enforcement runtime?

An LLM gateway primarily manages communication between applications and model providers. An enforcement runtime evaluates AI-generated messages against written rules before delivery. ZeroDrift's Enforcement Runtime returns a Pass, Fix, Block, or Escalate verdict and logs the decision as evidence. An LLM gateway and an enforcement runtime can work together, allowing companies to use both within the same AI application.

The Enforcement
Runtime for AI.

Every regulation, pre-loaded. Your policies, trained on top. Enforced on everything AI says, before it ships.