gateway
A Blazing Fast AI Gateway with integrated Guardrails. Route to 200+ LLMs, 50+ AI Guardrails with 1 fast & friendly API.
Top Related Projects
The official Python library for the OpenAI API
Integrate cutting-edge LLM technology quickly and easily into your apps
🦜🔗 Build context-aware reasoning applications
🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
Quick Overview
Portkey Gateway is an open-source AI gateway that provides a unified interface for multiple AI providers. It allows developers to easily switch between different AI models and providers, manage API keys, and handle rate limiting and retries. The project aims to simplify the integration of AI services into applications.
Pros
- Unified API for multiple AI providers (OpenAI, Anthropic, Azure, etc.)
- Easy switching between different AI models and providers
- Built-in rate limiting and retry mechanisms
- Customizable request/response handling
Cons
- Limited documentation for advanced features
- Relatively new project, may have undiscovered bugs
- Dependency on third-party AI providers
- Potential performance overhead due to the gateway layer
Code Examples
- Basic usage with OpenAI:
from portkey import Portkey
portkey = Portkey(api_key="your-api-key")
response = portkey.chat.completions.create(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "Hello, how are you?"}]
)
print(response.choices[0].message.content)
- Switching between providers:
from portkey import Portkey
portkey = Portkey(api_key="your-api-key")
# Using OpenAI
openai_response = portkey.chat.completions.create(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "Hello, OpenAI!"}]
)
# Using Anthropic
anthropic_response = portkey.chat.completions.create(
model="claude-2",
messages=[{"role": "user", "content": "Hello, Anthropic!"}]
)
- Custom request handling:
from portkey import Portkey
def custom_handler(request, response):
print(f"Request: {request}")
print(f"Response: {response}")
return response
portkey = Portkey(api_key="your-api-key", trace_id="custom-trace")
portkey.add_handler(custom_handler)
response = portkey.chat.completions.create(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "Hello, with custom handling!"}]
)
Getting Started
-
Install Portkey:
pip install portkey-ai -
Set up your API key:
from portkey import Portkey portkey = Portkey(api_key="your-api-key") -
Make your first API call:
response = portkey.chat.completions.create( model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hello, Portkey!"}] ) print(response.choices[0].message.content)
Competitor Comparisons
The official Python library for the OpenAI API
Pros of openai-python
- Official SDK for OpenAI API, ensuring direct compatibility and up-to-date features
- Comprehensive documentation and extensive community support
- Seamless integration with OpenAI's services and models
Cons of openai-python
- Limited to OpenAI's services, lacking support for other AI providers
- No built-in routing or load balancing capabilities
- Requires separate implementation for features like caching and rate limiting
Code Comparison
openai-python:
import openai
openai.api_key = "your-api-key"
response = openai.Completion.create(engine="davinci", prompt="Hello, world!")
print(response.choices[0].text)
gateway:
from portkey import Portkey
portkey = Portkey(api_key="your-api-key")
response = portkey.completions.create(model="gpt-3.5-turbo", prompt="Hello, world!")
print(response.choices[0].text)
The code snippets demonstrate that while openai-python is specifically tailored for OpenAI's services, gateway provides a more flexible interface that can potentially support multiple AI providers. gateway also offers additional features like caching, routing, and observability out of the box, making it a more comprehensive solution for managing AI API interactions.
Integrate cutting-edge LLM technology quickly and easily into your apps
Pros of Semantic Kernel
- More comprehensive framework for building AI applications
- Stronger integration with Azure and Microsoft ecosystem
- Extensive documentation and community support
Cons of Semantic Kernel
- Steeper learning curve due to its complexity
- Primarily focused on Microsoft technologies
- May be overkill for simple API routing needs
Code Comparison
Semantic Kernel (C#):
var kernel = Kernel.Builder.Build();
var function = kernel.CreateSemanticFunction("Generate a story about {{$input}}");
var result = await kernel.RunAsync("a brave knight", function);
Gateway (JavaScript):
const gateway = new Gateway();
gateway.addRoute('/generate', async (req, res) => {
const story = await openai.generateStory(req.body.input);
res.json({ story });
});
Summary
Semantic Kernel is a more comprehensive framework for building AI applications, with strong Microsoft ecosystem integration. However, it has a steeper learning curve and may be excessive for simple tasks. Gateway, on the other hand, focuses on API routing and management, making it more suitable for straightforward AI service integration but potentially less powerful for complex AI application development.
🦜🔗 Build context-aware reasoning applications
Pros of LangChain
- More comprehensive framework for building LLM applications
- Larger community and ecosystem with extensive documentation
- Supports a wider range of LLMs and integrations
Cons of LangChain
- Steeper learning curve due to its extensive features
- Can be overkill for simple LLM projects
- Less focused on API management and routing
Code Comparison
LangChain example:
from langchain import OpenAI, LLMChain, PromptTemplate
llm = OpenAI(temperature=0.9)
prompt = PromptTemplate(
input_variables=["product"],
template="What is a good name for a company that makes {product}?",
)
chain = LLMChain(llm=llm, prompt=prompt)
print(chain.run("colorful socks"))
Gateway example:
from gateway import Gateway
gateway = Gateway()
response = gateway.chat.completions.create(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "What is a good name for a company that makes colorful socks?"}]
)
print(response.choices[0].message.content)
The LangChain example showcases its abstraction layers and templating, while the Gateway example demonstrates its simpler, more direct approach to API calls.
🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
Pros of transformers
- Extensive library of pre-trained models for various NLP tasks
- Well-documented with comprehensive examples and tutorials
- Large and active community support
Cons of transformers
- Can be resource-intensive, especially for large models
- Steeper learning curve for beginners in NLP
- May require additional setup for specific hardware acceleration
Code Comparison
transformers:
from transformers import pipeline
classifier = pipeline("sentiment-analysis")
result = classifier("I love this product!")[0]
print(f"Label: {result['label']}, Score: {result['score']:.4f}")
gateway:
from portkey import Portkey
portkey = Portkey(api_key="your_api_key")
response = portkey.chat.completions.create(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "Analyze the sentiment: I love this product!"}]
)
print(response.choices[0].message.content)
The transformers library focuses on providing a wide range of pre-trained models for various NLP tasks, while gateway serves as an API gateway for multiple AI providers. transformers is more suitable for in-depth NLP work, while gateway simplifies access to different AI services through a unified interface.
Convert 
designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual CopilotREADME
The AI Gateway streamlines requests to 250+ language, vision, audio and image models with a unified API. It is production-ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency.
â
Blazing fast (9.9x faster) with a tiny footprint (~100kb build)
â
Load balance across multiple models, providers, and keys
â
Fallbacks make sure your app stays resilient
â
Automatic Retries with exponential fallbacks come by default
â
Configurable Request Timeouts to easily handle unresponsive LLM requests
â
Multimodal to support routing between Vision, TTS, STT, Image Gen, and more models
â
Plug-in middleware as needed
â
Battle tested over 480B tokens
â
Enterprise-ready for enhanced security, scale, and custom deployments
[!TIP] âï¸ Star this repo to get Github release notifications for new provider integrations and features.
Star History
Setup & Installation
Use the AI gateway through the hosted API or self-host the open-source or enterprise versions on your environment.
ð Hosted Gateway on portkey.ai (Fastest)
The hosted API is the fastest way to setup an AI Gateway for your Gen AI application. We process billions of tokens daily and is in production with companies like Postman, Haptik, Turing, MultiOn, SiteGPT, and more.
ð Self-hosting the OSS version (MIT License)
To run the AI gateway locally, execute the following command in your terminal. (Needs npx installed) Or, explore deployment guides for Cloudflare, Docker, Node.js and more here.
npx @portkey-ai/gateway
Your AI Gateway is now running on http://localhost:8787 ð
ð Self-hosting the Enterprise Version
The AI Gateway's enterprise version offers enterprise-ready capabilities for org management, governance, security and more out of the box. Compare the open source, hosted and enterprise versions here.
The enterprise deployment architecture, supported platforms is available here - Enterprise Private Cloud Deployments
Making requests through the AI gateway
Compatible with OpenAI API & SDKs
The AI Gateway is compatible with the OpenAI API & SDKs, and extends them to call 200+ LLMs reliably. To use the Gateway through OpenAI, update the client to include the gateway's URL and headers and make requests as usual. The AI gateway can translate requests written in the OpenAI format to the signature expected by the specified provider. View examples
Using the Python SDK 
Portkey Python SDK is a wrapper over the OpenAI Python SDK with added support for additional parameters across all other providers. If you're building with Python, this is the recommended library to connect to the Gateway.
pip install -qU portkey-ai
Using the Node.JS SDK
Portkey JS/TS SDK is a wrapper over the OpenAI JS SDK with added support for additional parameters across all other providers. If you're building with JS or TS, this is the recommended library to connect to the Gateway.
npm install --save portkey-ai
Using the REST APIs
The AI gateway supports OpenAI compatible endpoints with added parameter support for all other providers and models. View API Reference.
Other Integrations
| Language | Supported SDKs |
|---|---|
| JS / TS | LangchainJS LlamaIndex.TS |
| Python | Langchain LlamaIndex |
| Go | go-openai |
| Java | openai-java |
| Rust | async-openai |
| Ruby | ruby-openai |
Gateway Cookbooks
Trending Cookbooks
- Use models from Nvidia NIM with AI Gateway
- Monitor CrewAI Agents with Portkey!
- Comparing Top 10 LMSYS Models with AI Gateway.
Latest Cookbooks
- Create Synthetic Datasets using Nemotron
- Use Portkey Gateway with Vercel's AI SDK
- Monitor Llama Agents with Portkey
More Examples
Supported Providers
Explore Gateway integrations with 25+ providers and 6+ frameworks.
| Provider | Support | Stream | |
|---|---|---|---|
| OpenAI | â | â |
 | Azure OpenAI | â | â |
 | Anyscale | â | â |
 | Google Gemini & Palm | â | â |
 | Anthropic | â | â |
 | Cohere | â | â |
 | Together AI | â | â |
 | Perplexity | â | â |
 | Mistral | â | â |
 | Nomic | â | â |
 | AI21 | â | â |
 | Stability AI | â | â |
 | DeepInfra | â | â |
 | Ollama | â | â |
| Novita AI | â | â |
View the complete list of 200+ supported models here
Agents
Gateway seamlessly integrates with popular agent frameworks. Read the documentation here.
| Framework | Call 200+ LLMs | Advanced Routing | Caching | Logging & Tracing* | Observability* | Prompt Management* |
|---|---|---|---|---|---|---|
| Autogen | â | â | â | â | â | â |
| CrewAI | â | â | â | â | â | â |
| LangChain | â | â | â | â | â | â |
| Phidata | â | â | â | â | â | â |
| Llama Index | â | â | â | â | â | â |
| Control Flow | â | â | â | â | â | â |
| Build Your Own Agents | â | â | â | â | â | â |
*Only available on the hosted app. For detailed documentation click here.
Features
|
Fallbacks Fallback to another provider or model on failed requests. You can specify the errors on which to trigger the fallback. Improves reliability of your application 
|
Automatic Retries Automatically retry failed requests up to 5 times. An exponential backoff strategy spaces out retry attempts to prevent network overload. |
|
Load Balancing Distribute LLM requests across multiple API keys or AI providers with weights to ensure high availability and optimal performance. 
|
Request Timeouts Manage unruly LLMs & latencies by setting up granular request timeouts, allowing automatic termination of requests that exceed a specified duration. |
|
Multi-modal LLM Gateway Call vision, audio (text-to-speech & speech-to-text), and image generation models from multiple providers â all using the familiar OpenAI signature 
|
Guardrails Verify your LLM inputs AND outputs to adhere to your specified checks. Build your own checks or choose from the 20+ pre-built guardrails. 
|
These features are configured through the Gateway Config added to the x-portkey-config header or the config parameter in the SDKs.
Here's a sample config JSON showcasing the above features. All the features are optional
{
"retry": { "attempts": 5 },
"request_timeout": 10000,
"strategy": { "mode": "fallback" }, // or 'loadbalance', etc
"targets": [{
"provider": "openai",
"api_key": "sk-***"
},{
"strategy": {"mode": "loadbalance"}, // Optional nesting
"targets": {...}
}]
}
Then use the config in your API requests to the gateway.
Using Gateway Configs
Here's a guide to use the config object in your request.
Gateway Enterprise Version
Make your AI app more reliable and forward compatible, while ensuring complete data security and privacy.
â
Secure Key Management - for role-based access control and tracking
â
Simple & Semantic Caching - to serve repeat queries faster & save costs
â
Access Control & Inbound Rules - to control which IPs and Geos can connect to your deployments
â
PII Redaction - to automatically remove sensitive data from your requests to prevent indavertent exposure
â
SOC2, ISO, HIPAA, GDPR Compliances - for best security practices
â
Professional Support - along with feature prioritization
Schedule a call to discuss enterprise deployments
Contributing
The easiest way to contribute is to pick an issue with the good first issue tag ðª. Read the contribution guidelines here.
Bug Report? File here | Feature Request? File here
Community
Join our growing community around the world, for help, ideas, and discussions on AI.
Top Related Projects
The official Python library for the OpenAI API
Integrate cutting-edge LLM technology quickly and easily into your apps
🦜🔗 Build context-aware reasoning applications
🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
Convert designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual Copilot