Routing allows LangChain to dynamically select the best chain to handle a user’s query based on its content.
Let’s see it in action. Imagine we have a system that can answer questions about our company’s products and our company’s HR policies. We want to build a single entry point that figures out which of these two specialized systems to ask.
from langchain_core.runnables import RunnablePassthrough
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_community.chat_models import ChatOpenAI
from langchain.chains.router.router import MultiPromptChain
from langchain.chains.router.llm_router import LLMRouterChain, RouterOutputParser
from langchain.chains.router.multi_prompt_chain import ROUTER_TEMPLATE
# Assume you have these chains already defined:
# product_qa_chain = ...
# hr_policy_chain = ...
# For demonstration, let's create simple placeholder chains
llm = ChatOpenAI(temperature=0)
product_prompt = ChatPromptTemplate.from_template(
"You are a helpful assistant that answers questions about our products. \n\n{input}"
)
product_chain = (product_prompt | llm | StrOutputParser()).bind(tags=["product"])
hr_prompt = ChatPromptTemplate.from_template(
"You are a helpful assistant that answers questions about our HR policies. \n\n{input}"
)
hr_chain = (hr_prompt | llm | StrOutputParser()).bind(tags=["hr"])
# Define the destination chains and their corresponding descriptions
destination_chains = {
"product_qa": product_chain,
"hr_policy": hr_chain,
}
descriptions = {
"product_qa": "Questions about our company's products, features, and pricing.",
"hr_policy": "Questions about company HR policies, benefits, and employee guidelines.",
}
# Create the router chain
router_chain = LLMRouterChain.from_llm(
llm,
RouterOutputParser(),
# Use the ROUTER_TEMPLATE for the LLM to understand its task
# The template guides the LLM to choose a destination based on descriptions.
# It injects the descriptions and the input query for the LLM to process.
template=ROUTER_TEMPLATE,
# The LLM will output a JSON object with the chosen destination.
# The RouterOutputParser converts this JSON into a structured format.
output_parser=RouterOutputParser(),
# The input variables expected by the template.
input_variables=["input", "destination_chains"],
).bind(tags=["router"])
# Create the multi-prompt chain that orchestrates routing
full_chain = MultiPromptChain(
router_chain=router_chain,
destination_chains=destination_chains,
# The default chain to use if no specific destination is matched.
# Here, we'll just use the product chain as a fallback.
default_chain=product_chain,
# The key in the prompt that will be replaced by the descriptions.
# The LLM uses this to understand what each chain is for.
conditional_stop_strings=["\n\n"],
verbose=True,
).bind(tags=["multi_prompt"])
# Now, let's test it
query_product = "What are the key features of Product X?"
response_product = full_chain.invoke({"input": query_product})
print(f"Query: {query_product}\nResponse: {response_product}\n")
query_hr = "What is the company's policy on remote work?"
response_hr = full_chain.invoke({"input": query_hr})
print(f"Query: {query_hr}\nResponse: {response_hr}\n")
query_ambiguous = "Tell me about the company."
response_ambiguous = full_chain.invoke({"input": query_ambiguous})
print(f"Query: {query_ambiguous}\nResponse: {response_ambiguous}\n")
The core problem routing solves is efficiently directing user queries to the most appropriate specialized knowledge base or processing logic within a complex application. Instead of a single, monolithic LLM trying to be an expert in everything, we decompose the problem into smaller, focused "chains," each specialized in a particular domain. Routing is the mechanism that intelligently selects which of these specialized chains should handle a given input.
Internally, this is orchestrated by two main components: the LLMRouterChain and the MultiPromptChain. The LLMRouterChain is responsible for the decision-making. It takes the user’s input and a list of potential "destinations" (your specialized chains), along with descriptions of what each destination handles. It then uses an LLM, guided by a specific ROUTER_TEMPLATE, to determine which destination chain is the best fit for the query. The ROUTER_TEMPLATE is crucial; it instructs the LLM to act as a dispatcher, analyzing the input against the provided descriptions and outputting a choice.
The MultiPromptChain acts as the conductor. It holds the LLMRouterChain and a dictionary of all the available destination_chains. When invoked, it first passes the input to the router_chain. The router_chain determines the best destination_chain based on the input and descriptions. MultiPromptChain then takes that chosen destination and invokes the corresponding chain, passing the original input to it. If the router can’t confidently pick a destination, a default_chain can be specified to handle such cases, preventing a complete failure.
The exact levers you control are the destination_chains themselves (which are just LangChain Runnable objects, like a prompt-LLM-parser combo), the descriptions you provide for each chain (these are critical for the router LLM to make accurate decisions), and the ROUTER_TEMPLATE if you need to fine-tune the router’s behavior. The default_chain is also a key configuration for robustness.
The most surprising thing about routing is how much the quality of the descriptions matters, often more than the complexity of the underlying chains. The LLM routing is fundamentally a natural language understanding problem: can the LLM understand the user’s intent and map it to the description of a service? If your descriptions are vague, overlapping, or don’t accurately reflect the capabilities of the underlying chains, the router will make poor choices, even if the specialized chains themselves are perfect. A common mistake is to make descriptions too technical or too broad, failing to give the router LLM clear signals.
The next concept to explore is how to handle hierarchical routing, where a router might first select a broad category, and then a secondary router or logic further refines the choice within that category.