semantic-router
Superfast AI decision making and intelligent processing of multi-modal data.
Top Related Projects
🦜🔗 Build context-aware reasoning applications
Integrate cutting-edge LLM technology quickly and easily into your apps
Examples and guides for using the OpenAI API
:mag: AI orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots.
Quick Overview
Semantic Router is an open-source library that provides intelligent routing capabilities for large language models (LLMs). It aims to enhance the efficiency and accuracy of LLM-based applications by dynamically routing queries to the most appropriate model or tool based on the semantic content of the input.
Pros
- Improves efficiency by directing queries to the most suitable model or tool
- Enhances accuracy of LLM responses by leveraging specialized models for specific tasks
- Reduces computational costs by avoiding unnecessary processing of queries by inappropriate models
- Easily integrable with existing LLM-based applications
Cons
- May introduce additional latency due to the routing process
- Requires careful configuration and maintenance of routing rules
- Potential for misrouting if the semantic analysis is not accurate
- Limited documentation and examples available for advanced use cases
Code Examples
- Basic usage of Semantic Router:
from semantic_router import Router
from semantic_router.encoders import OpenAIEncoder
from semantic_router.routes import Route
router = Router(encoder=OpenAIEncoder())
routes = [
Route(name="greeting", utterances=["hello", "hi", "hey"]),
Route(name="farewell", utterances=["goodbye", "bye", "see you later"])
]
router.add_routes(routes)
result = router.route("Hello, how are you?")
print(result.name) # Output: greeting
- Using custom routes with functions:
from semantic_router import Router
from semantic_router.encoders import OpenAIEncoder
def handle_greeting(query):
return f"Hello! Nice to meet you. You said: {query}"
def handle_farewell(query):
return f"Goodbye! Have a great day. You said: {query}"
router = Router(encoder=OpenAIEncoder())
router.add_route("greeting", ["hello", "hi", "hey"], handle_greeting)
router.add_route("farewell", ["goodbye", "bye", "see you later"], handle_farewell)
result = router.route("Hi there!")
print(result.function("Hi there!")) # Output: Hello! Nice to meet you. You said: Hi there!
- Using Semantic Router with async functions:
import asyncio
from semantic_router import Router
from semantic_router.encoders import OpenAIEncoder
async def async_handle_greeting(query):
await asyncio.sleep(1) # Simulating async operation
return f"Hello! Nice to meet you. You said: {query}"
router = Router(encoder=OpenAIEncoder())
router.add_route("greeting", ["hello", "hi", "hey"], async_handle_greeting)
async def main():
result = await router.aroute("Hello, how are you?")
response = await result.function("Hello, how are you?")
print(response)
asyncio.run(main())
Getting Started
To get started with Semantic Router, follow these steps:
-
Install the library:
pip install semantic-router
-
Import and initialize the Router:
from semantic_router import Router from semantic_router.encoders import OpenAIEncoder router = Router(encoder=OpenAIEncoder())
-
Add routes and start routing queries:
router.add_route("greeting", ["hello", "hi", "hey"]) router.add_route("farewell", ["goodbye", "bye", "see you later"]) result = router.route("Hello, how are you?") print(result.name) # Output: greeting
Competitor Comparisons
🦜🔗 Build context-aware reasoning applications
Pros of LangChain
- Comprehensive framework with a wide range of tools and integrations
- Extensive documentation and active community support
- Flexible architecture allowing for customization and extension
Cons of LangChain
- Steeper learning curve due to its extensive feature set
- Potential overhead for simpler projects that don't require all functionalities
- May require more setup and configuration compared to lightweight alternatives
Code Comparison
Semantic Router:
from semantic_router import Route, RouteLayer
routes = [
Route(name="greeting", utterances=["hello", "hi", "hey"]),
Route(name="farewell", utterances=["goodbye", "bye", "see you"])
]
layer = RouteLayer(routes)
result = layer("Hello there!")
LangChain:
from langchain.prompts import PromptTemplate
from langchain.llms import OpenAI
prompt = PromptTemplate.from_template("Say {text}")
llm = OpenAI()
chain = prompt | llm
result = chain.invoke({"text": "Hello there!"})
Both libraries offer ways to process and route text, but LangChain provides a more comprehensive set of tools for building complex language model applications, while Semantic Router focuses specifically on semantic routing functionality.
Integrate cutting-edge LLM technology quickly and easily into your apps
Pros of Semantic Kernel
- More comprehensive framework with broader functionality
- Extensive documentation and community support
- Integration with various AI services and models
Cons of Semantic Kernel
- Steeper learning curve due to its complexity
- Potentially overkill for simpler projects
- Tighter coupling with Microsoft ecosystem
Code Comparison
Semantic Router:
from semantic_router import Route
route = Route(
name="weather",
utterances=["What's the weather like?", "Is it going to rain today?"]
)
Semantic Kernel:
using Microsoft.SemanticKernel;
var kernel = Kernel.Builder.Build();
var weatherSkill = kernel.ImportSkill(new WeatherSkill());
var result = await kernel.RunAsync("What's the weather like?", weatherSkill["GetWeather"]);
The Semantic Router focuses on routing intents, while Semantic Kernel provides a more comprehensive framework for building AI-powered applications. Semantic Router offers a simpler approach for specific use cases, whereas Semantic Kernel provides more flexibility and integration options at the cost of increased complexity.
Examples and guides for using the OpenAI API
Pros of openai-cookbook
- Comprehensive collection of examples and best practices for using OpenAI's APIs
- Covers a wide range of use cases and techniques, from basic to advanced
- Regularly updated with new features and improvements from OpenAI
Cons of openai-cookbook
- Focuses solely on OpenAI's offerings, limiting its applicability to other AI platforms
- May be overwhelming for beginners due to its extensive content
- Lacks a specific focus on routing or intent classification
Code Comparison
semantic-router:
from semantic_router import Route
route = Route(
name="greeting",
utterances=["hi", "hello", "hey"],
function=lambda x: "Hello there!"
)
openai-cookbook:
import openai
response = openai.Completion.create(
engine="text-davinci-002",
prompt="Translate the following English text to French: 'Hello, how are you?'",
max_tokens=60
)
The semantic-router example demonstrates a simple route definition for intent classification, while the openai-cookbook example shows how to use OpenAI's API for text completion tasks. semantic-router is more focused on routing and intent classification, whereas openai-cookbook provides a broader range of examples for various AI tasks using OpenAI's services.
:mag: AI orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots.
Pros of Haystack
- More comprehensive framework for building end-to-end NLP applications
- Larger community and ecosystem with extensive documentation
- Supports multiple languages and integrates with various NLP models
Cons of Haystack
- Steeper learning curve due to its extensive features
- Potentially overkill for simpler routing tasks
- Heavier resource requirements for deployment
Code Comparison
Semantic-router:
from semantic_router import Route, RouteLayer
routes = [
Route(name="greeting", utterances=["hi", "hello"]),
Route(name="farewell", utterances=["bye", "goodbye"])
]
layer = RouteLayer(routes)
result = layer("hello there")
Haystack:
from haystack import Pipeline
from haystack.nodes import TextConverter, Preprocessor, FARMReader
pipeline = Pipeline()
pipeline.add_node(component=TextConverter(), name="TextConverter", inputs=["File"])
pipeline.add_node(component=Preprocessor(), name="Preprocessor", inputs=["TextConverter"])
pipeline.add_node(component=FARMReader(model_name_or_path="deepset/roberta-base-squad2"), name="Reader", inputs=["Preprocessor"])
Summary
Semantic-router is more focused on lightweight semantic routing, while Haystack offers a comprehensive NLP pipeline framework. Semantic-router is easier to set up for simple routing tasks, whereas Haystack provides more advanced features but with increased complexity.
Convert designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual CopilotREADME
Semantic Router is a superfast decision-making layer for your LLMs and agents. Rather than waiting for slow LLM generations to make tool-use decisions, we use the magic of semantic vector space to make those decisions â routing our requests using semantic meaning.
Read the Docs
Quickstart
To get started with semantic-router we install it like so:
pip install -qU semantic-router
âï¸ If wanting to use a fully local version of semantic router you can use HuggingFaceEncoder
and LlamaCppLLM
(pip install -qU "semantic-router[local]"
, see here). To use the HybridRouteLayer
you must pip install -qU "semantic-router[hybrid]"
.
We begin by defining a set of Route
objects. These are the decision paths that the semantic router can decide to use, let's try two simple routes for now â one for talk on politics and another for chitchat:
from semantic_router import Route
# we could use this as a guide for our chatbot to avoid political conversations
politics = Route(
name="politics",
utterances=[
"isn't politics the best thing ever",
"why don't you tell me about your political opinions",
"don't you just love the president",
"they're going to destroy this country!",
"they will save the country!",
],
)
# this could be used as an indicator to our chatbot to switch to a more
# conversational prompt
chitchat = Route(
name="chitchat",
utterances=[
"how's the weather today?",
"how are things going?",
"lovely weather today",
"the weather is horrendous",
"let's go to the chippy",
],
)
# we place both of our decisions together into single list
routes = [politics, chitchat]
We have our routes ready, now we initialize an embedding / encoder model. We currently support a CohereEncoder
and OpenAIEncoder
â more encoders will be added soon. To initialize them we do:
import os
from semantic_router.encoders import CohereEncoder, OpenAIEncoder
# for Cohere
os.environ["COHERE_API_KEY"] = "<YOUR_API_KEY>"
encoder = CohereEncoder()
# or for OpenAI
os.environ["OPENAI_API_KEY"] = "<YOUR_API_KEY>"
encoder = OpenAIEncoder()
With our routes
and encoder
defined we now create a RouteLayer
. The route layer handles our semantic decision making.
from semantic_router.layer import RouteLayer
rl = RouteLayer(encoder=encoder, routes=routes)
We can now use our route layer to make super fast decisions based on user queries. Let's try with two queries that should trigger our route decisions:
rl("don't you love politics?").name
[Out]: 'politics'
Correct decision, let's try another:
rl("how's the weather today?").name
[Out]: 'chitchat'
We get both decisions correct! Now lets try sending an unrelated query:
rl("I'm interested in learning about llama 2").name
[Out]:
In this case, no decision could be made as we had no matches â so our route layer returned None
!
Integrations
The encoders of semantic router include easy-to-use integrations with Cohere, OpenAI, Hugging Face, FastEmbed, and more â we even support multi-modality!.
Our utterance vector space also integrates with Pinecone and Qdrant!
ð Resources
Docs
Notebook | Description |
---|---|
Introduction | Introduction to Semantic Router and static routes |
Dynamic Routes | Dynamic routes for parameter generation and functionc calls |
Save/Load Layers | How to save and load RouteLayer from file |
LangChain Integration | How to integrate Semantic Router with LangChain Agents |
Local Execution | Fully local Semantic Router with dynamic routes â local models such as Mistral 7B outperform GPT-3.5 in most tests |
Route Optimization | How to train route layer thresholds to optimize performance |
Multi-Modal Routes | Using multi-modal routes to identify Shrek vs. not-Shrek pictures |
Online Course
Community
- Dimitrios Manias, Ali Chouman, Abdallah Shami, Semantic Routing for Enhanced Performance of LLM-Assisted Intent-Based 5G Core Network Management and Orchestration, IEEE GlobeCom 2024
- Julian Horsey, Semantic Router superfast decision layer for LLMs and AI agents, Geeky Gadgets
- azhar, Beyond Basic Chatbots: How Semantic Router is Changing the Game, AI Insights @ Medium
- Daniel Avila, Semantic Router: Enhancing Control in LLM Conversations, CodeGPT @ Medium
- Yogendra Sisodia, Stop Chat-GPT From Going Rogue In Production With Semantic Router, Medium
- Aniket Hingane, LLM Apps: Why you Must Know Semantic Router in 2024: Part 1, Medium
- Adrien Sales, ð Semantic Router w. ollama/gemma2 : real life 10ms hotline challenge ð¤¯
- Adrien Sales, Kaggle Notebook ð Semantic Router:
ollama
/gemma2:9b
hotline
Top Related Projects
🦜🔗 Build context-aware reasoning applications
Integrate cutting-edge LLM technology quickly and easily into your apps
Examples and guides for using the OpenAI API
:mag: AI orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots.
Convert designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual Copilot