LlamaIndex’s Router isn’t just a fancy if/else for your queries; it’s a dynamic dispatch system that can reroute a single natural language question to the most appropriate specialized knowledge base, even if that knowledge base is a completely different type of index.
Let’s see it in action. Imagine you have two distinct data sources: one holding general company FAQs and another containing detailed product specifications.
from llama_index.core import VectorStoreIndex, SummaryIndex, Settings
from llama_index.core.indices.query.query_transform.base import DecomposeQueryTransform
from llama_index.core.tools import QueryEngineTool, ToolMetadata
from llama_index.core.objects import ObjectStore
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.prompts import PromptTemplate
from llama_index.core.response.notebook_utils import display_response
from llama_index.core.query_engine import RouterQueryEngine
from llama_index.core.selectors import LLMSingleSelector
from llama_index.core.schema import TextNode
from llama_index.core.storage import InMemoryStorage
from llama_index.core.vector_store import VectorStore
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI
import os
# --- Configuration ---
# Ensure you have your OpenAI API key set as an environment variable
# os.environ["OPENAI_API_KEY"] = "YOUR_API_KEY"
Settings.llm = OpenAI(model="gpt-3.5-turbo")
Settings.embed_model = OpenAIEmbedding()
Settings.node_parser = SentenceSplitter(chunk_size=1024, chunk_overlap=20)
Settings.transformations = [Settings.node_parser]
# --- Data Setup ---
# FAQ Data
faq_text = """
Q: How do I reset my password?
A: To reset your password, go to the login page and click the "Forgot Password" link.
You will receive an email with instructions.
Q: What are your business hours?
A: Our business hours are Monday to Friday, 9 AM to 5 PM EST.
Q: How can I contact customer support?
A: You can contact customer support by emailing support@example.com or by calling us at 1-800-123-4567.
"""
# Product Specs Data
product_specs_text = """
Product: Quantum Widget X1
Model Number: QWX1-2023
Specifications:
- Dimensions: 10cm x 5cm x 2cm
- Weight: 150g
- Power Source: 5V DC adapter (included)
- Connectivity: Bluetooth 5.0, Wi-Fi 802.11n
- Operating Temperature: 0°C to 40°C
- Warranty: 1 year limited
Product: Galactic Gizmo Z2
Model Number: GGZ2-PRO
Specifications:
- Dimensions: 15cm x 8cm x 3cm
- Weight: 250g
- Power Source: USB-C, 12V DC adapter (sold separately)
- Connectivity: Wi-Fi 802.11ac, Ethernet
- Operating Temperature: -10°C to 50°C
- Warranty: 2 year limited
"""
# --- Index Creation ---
# FAQ Index (using SummaryIndex for high-level answers)
faq_nodes = Settings.node_parser.get_nodes_from_documents([TextNode(text=faq_text)])
faq_index = SummaryIndex(faq_nodes)
faq_engine = faq_index.as_query_engine(
response_prompt=PromptTemplate(
"You are an AI assistant for general company FAQs. \n"
"Here is the relevant context: {context_str}\n"
"Q: {query_str}\nA:"
)
)
# Product Specs Index (using VectorStoreIndex for detailed lookups)
product_nodes = Settings.node_parser.get_nodes_from_documents([TextNode(text=product_specs_text)])
product_index = VectorStoreIndex(product_nodes)
product_engine = product_index.as_query_engine(
response_prompt=PromptTemplate(
"You are an AI assistant for product specifications. \n"
"Here is the relevant context: {context_str}\n"
"Q: {query_str}\nA:"
)
)
# --- Router Setup ---
# Define tools for each index
faq_tool = QueryEngineTool(
query_engine=faq_engine,
metadata=ToolMetadata(
name="faq_tool",
description="Provides answers to general company FAQs, such as business hours, password resets, and contact information."
),
)
product_tool = QueryEngineTool(
query_engine=product_engine,
metadata=ToolMetadata(
name="product_tool",
description="Provides detailed specifications for products like Quantum Widget X1 and Galactic Gizmo Z2, including model numbers, dimensions, power, and warranty."
),
)
# Create the RouterQueryEngine
router_engine = RouterQueryEngine(
selector=LLMSingleSelector(), # Selects the best tool
query_engine_tools=[
faq_tool,
product_tool,
],
)
# --- Querying ---
print("--- Querying FAQ ---")
response_faq = router_engine.query("What are your business hours?")
display_response(response_faq)
print("\n--- Querying Product Specs ---")
response_specs = router_engine.query("What is the weight of the Quantum Widget X1?")
display_response(response_specs)
print("\n--- Router Deciding ---")
response_decision = router_engine.query("How do I reset my password for the Quantum Widget X1?")
display_response(response_decision)
The magic happens in RouterQueryEngine. You give it a list of QueryEngineTool objects, each with a ToolMetadata that includes a name and a description. The RouterQueryEngine uses an LLM (specified by selector, here LLMSingleSelector which picks the single best tool) to read your query and the descriptions of your tools. It then decides which tool is most relevant and forwards the query only to that tool’s query_engine.
For instance, asking "What are your business hours?" will be routed to the faq_tool because its description clearly covers FAQs. Asking "What is the weight of the Quantum Widget X1?" will hit the product_tool. The truly impressive part is when the query is ambiguous, like "How do I reset my password for the Quantum Widget X1?". The router, based on the descriptions, will likely pick the faq_tool because password resets are a general FAQ, not a product-specific spec.
The selector parameter is key. LLMSingleSelector is the most common, aiming to pick just one tool. Other selectors exist, like LLMMultiSelector to query multiple tools, or custom ones for more complex routing logic. The response_prompt within each tool’s query_engine is also crucial for guiding the LLM’s behavior when it does use that tool.
The system solves the problem of managing multiple, distinct knowledge bases without needing a complex orchestration layer. Instead of writing code to parse queries and manually select an index, you define your data sources as tools, describe them, and let the router handle the dispatch. This allows for scalability as you add more specialized indices.
Most people understand that the router picks a tool based on its description. What they often miss is how the response_prompt within each individual tool’s query engine can dramatically influence the final answer, even after the router has made its selection. If the faq_tool’s response_prompt is poorly written, it might still produce a bad answer even if the router correctly identified it as the right tool. The router is the gatekeeper, but the individual query engine is the one that does the actual work and can still fumble.
The next step is exploring how to make the router consider multiple tools for a single query using LLMMultiSelector.