LlamaIndex actually uses a copy of your data for indexing, not a direct reference, which is why you can modify or delete the original source files without breaking your queryability.
Let’s see LlamaIndex in action. Imagine you have a directory of PDFs you want to query.
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
# Load documents from a directory
documents = SimpleDirectoryReader("./my_pdfs").load_data()
# Create an index from the documents
index = VectorStoreIndex.from_documents(documents)
# Query the index
query_engine = index.as_query_engine()
response = query_engine.query("What is the main topic of these documents?")
print(response)
This SimpleDirectoryReader is the first step in our ingestion pipeline. It’s responsible for reading data from various sources. For directories, it recursively scans for supported file types (PDFs, text files, etc.). Each file is parsed into a Document object, which is LlamaIndex’s fundamental unit of data. This object contains the text content and can also hold metadata like file_path or page_label.
The VectorStoreIndex.from_documents(documents) is where the magic of indexing happens. LlamaIndex takes your Document objects and:
- Splits them: Large documents are broken down into smaller chunks. This is crucial because embedding models have token limits, and smaller chunks lead to more focused embeddings. You can control this splitting with
TextSplitterobjects. - Embeds them: Each chunk is converted into a numerical vector (an embedding) using an embedding model (like OpenAI’s
text-embedding-ada-002or an open-source one). These vectors capture the semantic meaning of the text. - Stores them: The text chunks and their corresponding embeddings are stored in a
VectorStore. By default, LlamaIndex uses an in-memorySimpleVectorStore, but for production, you’d connect to dedicated vector databases like Pinecone, Weaviate, or Chroma.
The as_query_engine() method then creates an interface for interacting with your indexed data. When you ask a question, the query engine:
- Embeds your query: Your question is also converted into an embedding vector.
- Performs similarity search: It searches the
VectorStorefor text chunks whose embeddings are most similar (closest in vector space) to your query embedding. - Synthesizes an answer: The retrieved text chunks are passed to a Large Language Model (LLM) along with your original question, and the LLM generates a coherent answer based on the provided context.
The real power comes in configuring this pipeline. You can swap out the document loader, the text splitter, the embedding model, the LLM, and the vector store. For instance, to use a local embedding model and a more robust splitter:
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.llms.openai import OpenAI # Or other LLM
# Configure global settings
Settings.llm = OpenAI(model="gpt-3.5-turbo")
Settings.embed_model = HuggingFaceEmbedding(model_name="sentence-transformers/all-MiniLM-L6-v2")
Settings.node_parser = SentenceSplitter(chunk_size=1024, chunk_overlap=20)
# Load documents
documents = SimpleDirectoryReader("./my_pdfs").load_data()
# Create index (uses global settings)
index = VectorStoreIndex.from_documents(documents)
# Query
query_engine = index.as_query_engine()
response = query_engine.query("What are the key performance indicators mentioned?")
print(response)
Here, we’ve explicitly set Settings.embed_model to use a local HuggingFace model, which can be faster and more cost-effective than API calls. We’ve also configured Settings.node_parser to use SentenceSplitter with a chunk_size of 1024 tokens and chunk_overlap of 20 tokens. The chunk_overlap is critical; it ensures that context isn’t lost at the boundaries of chunks. If a sentence straddles two chunks, the overlap ensures the full sentence and its surrounding context are available to the embedding model for both chunks, improving retrieval accuracy.
The most surprising aspect of LlamaIndex’s ingestion is how it handles the relationship between your source data and the indexed data. When you call VectorStoreIndex.from_documents(), LlamaIndex doesn’t just store pointers to your original files. It parses the content, splits it into nodes, embeds these nodes, and stores both the text of the nodes and their embeddings in the vector store. Your original documents are effectively duplicated and transformed into a format optimized for semantic search. This means you can safely delete, move, or modify the original files after indexing without affecting the searchability of the data that has already been ingested into the index. The index becomes its own independent, searchable knowledge base.
Beyond basic document loading, LlamaIndex offers specialized readers for databases (SQL, NoSQL), APIs (Slack, Notion, Google Drive), and even web pages. Each reader abstracts away the complexities of fetching data and presents it as a list of Document objects, fitting seamlessly into the indexing pipeline.
The next hurdle is managing and updating indexes as your source data evolves.