Loaders and Splitters

Load PDFs and web pages, then split them so retrieval stays accurate.

• Intermediate
• 16 min read
• 3 objectives

Before this lessonLesson 6: Chat Memory

RAG quality is usually won or lost before the model: what you load, and how you cut it into chunks.

Loaders

from langchain_community.document_loaders import PyPDFLoader, WebBaseLoader, TextLoader

pages = PyPDFLoader("handbook.pdf").load()          # one Document per page
web = WebBaseLoader("https://example.com/docs").load()
notes = TextLoader("notes.md", encoding="utf-8").load()

Each Document has page_content and metadata (source path, page number). Keep that metadata: you will cite it later.

Splitting

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=120,
    separators=["\n## ", "\n\n", "\n", " ", ""],
)
chunks = splitter.split_documents(pages)
print(len(chunks), chunks[0].metadata)

• Too large: retrieval returns a blob the model skims badly.
• Too small: a sentence without its heading is meaningless.
• Overlap keeps a sentence that straddles a cut in both chunks.