langchain-text-splitters reference

Pinned version: langchain-text-splitters==1.1.2 (released 2026-04-16; it is the version pip resolves beside langchain-core==1.6.2, and its only requirement is langchain-core<2.0.0,>=1.2.31). Everything below was derived by inspecting this exact installed package, and every example was executed against it on Python 3.13. Each heading links to reference.langchain.com, which is unversioned and always serves the latest 1.x. Links open in a new tab.

Part of a five-page set — see how the LangChain stack fits together for how langchain-core, langgraph, langchain and langchain-text-splitters fit together.

What langchain-text-splitters is

• One job: cut long text into chunks that fit a budget. An embedding model, a context window and a retrieval hit all want pieces of bounded size that still read as a unit — a paragraph, a function, a section. Every class here turns one string (or one Document) into a list of them, and langchain-core's Document metadata is copied onto every chunk.
• A splitter is a BaseDocumentTransformer. The base class, TextSplitter, subclasses that langchain-core interface, so a splitter can sit anywhere a document transformer is expected; BaseLoader.load_and_split is the common call site.
• It depends on langchain-core and nothing else in the stack. Importing it leaves both langgraph and langchain absent from sys.modules; it sits beside the agent stack, not in it.
• The 0.x import path is gone. from langchain.text_splitter import … was moved to this package in 0.2 and does not exist in langchain 1.4.0 — a tutorial using it predates the split.

The classes fall into four families; only the first two run with what is installed here:

The token-counting family matters in production — an embedding model's limit is in tokens, not characters — but it changes only length_function, not the algorithm, so everything below about chunk_size and chunk_overlap applies to it unchanged.

Index

TextSplitter

The abstract base of every size-bounded splitter. It leaves exactly one member abstract, split_text(text: str) -> list[str], and its constructor fixes the budget every subclass shares:

• chunk_size=4000 — the upper bound, measured by length_function (default len, so characters; the from_tiktoken_encoder/from_huggingface_tokenizer classmethods swap in a token count). chunk_overlap=200 — how much of the end of one chunk is repeated at the start of the next, so a sentence cut at a boundary survives in one of them; it must be smaller than chunk_size.
• keep_separator=False — whether the separator a subclass split on stays in the pieces (True/"start" attaches it to the following piece, "end" to the preceding one); strip_whitespace=True trims each chunk; add_start_index=False records each chunk's offset into the original text as metadata["start_index"].

What it provides on top of split_text: create_documents(texts, metadatas=None) splits each text and wraps every chunk in a Document carrying a deep copy of that text's metadata; split_documents(docs) does the same for existing documents; transform_documents is the BaseDocumentTransformer entry point and calls split_documents. The packing of small pieces into chunks of chunk_size with chunk_overlap lives in the protected _merge_splits(pieces, separator), which is what the shipped subclasses call after cutting — the subclass below does the same with sentences.

 1import re
 2
 3from langchain_core.documents import Document
 4from langchain_core.documents import BaseDocumentTransformer
 5from langchain_text_splitters import TextSplitter
 6
 7
 8class SentenceSplitter(TextSplitter):
 9    """split_text is the only abstract member; the base class turns it into documents."""
10
11    def split_text(self, text: str) -> list[str]:
12        sentences = re.split(r"(?<=[.!?])\s+", text)
13        return self._merge_splits(sentences, " ")      # packs sentences into chunk_size, with overlap
14
15
16text = "Cats purr. Cats also sleep a lot. Dogs bark. Dogs fetch sticks. Fish are quiet."
17splitter = SentenceSplitter(chunk_size=40, chunk_overlap=20, add_start_index=True)
18print(isinstance(splitter, BaseDocumentTransformer))                 # True
19
20for chunk in splitter.split_text(text):
21    print(len(chunk), "|", chunk)
22# 33 | Cats purr. Cats also sleep a lot.
23# 29 | Dogs bark. Dogs fetch sticks.
24# 34 | Dogs fetch sticks. Fish are quiet.   <- overlap: the last sentence is repeated
25
26docs = splitter.create_documents([text], metadatas=[{"source": "pets.txt"}])
27print(docs[1].metadata)                                              # {'source': 'pets.txt', 'start_index': 34}
28
29again = splitter.split_documents([Document(text, metadata={"source": "pets.txt"})])
30print(len(again), again[0].metadata)                                 # 3 {'source': 'pets.txt', 'start_index': 0}

↩ back to index

CharacterTextSplitter

The simplest concrete splitter: one separator, one pass. split_text cuts the text at every occurrence of separator (default "\n\n", so paragraphs; a regular expression when is_separator_regex=True), drops empty pieces, then packs consecutive pieces into chunks: a piece is added while the running total stays within chunk_size, otherwise the current chunk is emitted (joined with the separator, so "\n\n" reappears between paragraphs that share a chunk) and the trailing pieces that fit in chunk_overlap are carried into the next one.

Two consequences to know: a single piece longer than chunk_size is never cut — it becomes a chunk on its own and the splitter logs Created a chunk of size N, which is longer than the specified M — and with a one-character separator like " " the overlap is measured in pieces, so the same words show up at the end of one chunk and the start of the next. When you need long paragraphs cut down too, use RecursiveCharacterTextSplitter .

 1from langchain_text_splitters import CharacterTextSplitter
 2
 3text = "Cats purr.\n\nCats also sleep a lot.\n\nDogs bark. Dogs fetch sticks.\n\nFish are quiet."
 4
 5splitter = CharacterTextSplitter(separator="\n\n", chunk_size=40, chunk_overlap=0)
 6for chunk in splitter.split_text(text):
 7    print(len(chunk), "|", repr(chunk))
 8# 34 | 'Cats purr.\n\nCats also sleep a lot.'
 9# 29 | 'Dogs bark. Dogs fetch sticks.'
10# 15 | 'Fish are quiet.'
11
12# The separator is the only cut point: a paragraph longer than chunk_size passes through whole.
13small = CharacterTextSplitter(separator="\n\n", chunk_size=20, chunk_overlap=0)
14print([len(c) for c in small.split_text(text)])           # [10, 22, 29, 15]   <- and the warning on stderr
15
16words = CharacterTextSplitter(separator=" ", chunk_size=12, chunk_overlap=5)
17print(words.split_text("Cats purr. Dogs bark. Fish are quiet."))
18# ['Cats purr.', 'purr. Dogs', 'Dogs bark.', 'bark. Fish', 'Fish are', 'are quiet.']   <- chunk_overlap re-uses trailing pieces

↩ back to index

RecursiveCharacterTextSplitter

The default choice, and what load_and_split() uses when given nothing. It takes a list of separators — by default ["\n\n", "\n", " ", ""] — and works down it: the text is cut on the first separator that occurs in it; pieces that fit are packed into chunks as in CharacterTextSplitter; a piece that is still longer than chunk_size is split again with the next separator, and so on down to single characters. The result is that each chunk is the largest natural unit that fits — whole paragraphs where they are short enough, otherwise lines, otherwise runs of words — which is why the size-20 run below cuts one sentence at a space and leaves the others intact. keep_separator defaults to True here (the separator stays at the start of the piece that follows it), so nothing is lost between chunks.

For source code and markup, from_language(Language.X, **kwargs) builds one with a separator list that starts at that language's structural boundaries — Language is an Enum of 28 names (PYTHON, MARKDOWN, JS, GO, RUST, HTML, LATEX, …) and get_separators_for_language(lang) shows the list it would use. PythonCodeTextSplitter, MarkdownTextSplitter and LatexTextSplitter are exactly these presets as subclasses.

 1from langchain_text_splitters import Language, RecursiveCharacterTextSplitter
 2
 3text = "Cats purr.\nCats also sleep a lot.\n\nDogs bark. Dogs fetch sticks.\n\nFish are quiet."
 4
 5splitter = RecursiveCharacterTextSplitter(chunk_size=25, chunk_overlap=0)
 6print(splitter._separators)                             # ['\n\n', '\n', ' ', '']
 7for chunk in splitter.split_text(text):
 8    print(len(chunk), "|", repr(chunk))
 9# 10 | 'Cats purr.'
10# 22 | 'Cats also sleep a lot.'
11# 21 | 'Dogs bark. Dogs fetch'     <- no newline inside this paragraph, so it fell back to spaces
12# 7 | 'sticks.'
13# 15 | 'Fish are quiet.'
14
15code = '''def add(a, b):
16    return a + b
17
18
19class Greeter:
20    def hello(self):
21        return "hi"
22'''
23py = RecursiveCharacterTextSplitter.from_language(Language.PYTHON, chunk_size=60, chunk_overlap=0)
24print(RecursiveCharacterTextSplitter.get_separators_for_language(Language.PYTHON))
25# ['\nclass ', '\ndef ', '\n\tdef ', '\n\n', '\n', ' ', '']
26for chunk in py.split_text(code):
27    print(repr(chunk))
28# 'def add(a, b):\n    return a + b'
29# 'class Greeter:\n    def hello(self):\n        return "hi"'
30print(len(Language), Language.MARKDOWN.value)           # 28 markdown

↩ back to index

MarkdownHeaderTextSplitter

A structure splitter rather than a size splitter: it is not a TextSplitter subclass, has no chunk_size, and returns Documents from split_text. You pass headers_to_split_on, a list of (marker, metadata_key) pairs such as [("#", "h1"), ("##", "h2")]; it walks the text line by line, keeps track of the current heading at each level, and emits one Document per stretch of text under the same headings, with those headings in metadata under the keys you named (lines inside fenced code blocks are never taken as headings). So a chunk always knows which section of which chapter it came from — the thing a plain size split loses.

Options: strip_headers=True removes the heading lines from page_content (set it to False to keep them, as the second call below shows); return_each_line=True emits one Document per line instead of per section; custom_header_patterns={"**": 1} treats bold-only lines as headings of the given level. Because there is no size bound, the usual pattern is two passes: this splitter first, then a RecursiveCharacterTextSplitter.split_documents over its output — the size splitter copies the heading metadata onto every chunk it makes.

 1from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter
 2
 3md = """# Pets
 4
 5Intro line.
 6
 7## Cats
 8
 9Cats purr.
10Cats sleep.
11
12## Dogs
13
14Dogs bark.
15"""
16
17splitter = MarkdownHeaderTextSplitter(headers_to_split_on=[("#", "h1"), ("##", "h2")])
18for doc in splitter.split_text(md):
19    print(doc.metadata, "|", repr(doc.page_content))
20# {'h1': 'Pets'} | 'Intro line.'
21# {'h1': 'Pets', 'h2': 'Cats'} | 'Cats purr.\nCats sleep.'
22# {'h1': 'Pets', 'h2': 'Dogs'} | 'Dogs bark.'
23
24keep = MarkdownHeaderTextSplitter(headers_to_split_on=[("##", "h2")], strip_headers=False)
25print(repr(keep.split_text(md)[1].page_content))        # '## Cats  \nCats purr.\nCats sleep.'
26
27# Second pass: a size-based splitter keeps the header metadata on every piece.
28chunks = RecursiveCharacterTextSplitter(chunk_size=12, chunk_overlap=0).split_documents(splitter.split_text(md))
29print([(c.metadata.get("h2"), c.page_content) for c in chunks])
30# [(None, 'Intro line.'), ('Cats', 'Cats purr.'), ('Cats', 'Cats sleep.'), ('Dogs', 'Dogs bark.')]

↩ back to index

RecursiveJsonSplitter

For data rather than prose: it splits a nested dict into smaller dicts without breaking the structure. It walks the keys depth-first, adding each key: value to the current chunk while the chunk's serialised size (len(json.dumps(chunk))) stays under max_chunk_size (default 2000); when the next item would not fit and the chunk already holds at least min_chunk_size (default max_chunk_size - 200, floored at 50), it starts a new chunk — and descends into the value, so a large sub-object is spread over several chunks that each repeat the key path above it. Every chunk is therefore valid JSON with the original nesting. The size is a target, not a hard cap: a value is admitted when its own size fits the remaining room, and the nesting it lands in can push the chunk a few characters over, as the second chunk below shows.

Lists are not split unless convert_lists=True, which first rewrites them as dicts keyed by index. Three outputs: split_json(data) → list of dicts, split_text(data, ensure_ascii=True) → the same as JSON strings, create_documents(texts=[data, …], metadatas=…) → Documents.

 1import json
 2
 3from langchain_text_splitters import RecursiveJsonSplitter
 4
 5data = {
 6    "cats": {"sleep": "sixteen hours", "sound": "purr", "food": "fish"},
 7    "dogs": {"sleep": "twelve hours", "sound": "bark"},
 8    "fish": {"sound": "none"},
 9}
10
11splitter = RecursiveJsonSplitter(max_chunk_size=60)
12for chunk in splitter.split_json(data):
13    print(len(json.dumps(chunk)), "|", chunk)
14# 53 | {'cats': {'sleep': 'sixteen hours', 'sound': 'purr'}}
15# 61 | {'cats': {'food': 'fish'}, 'dogs': {'sleep': 'twelve hours'}}   <- the "cats" path is repeated
16# 54 | {'dogs': {'sound': 'bark'}, 'fish': {'sound': 'none'}}
17
18print(splitter.split_text(data)[0])                     # {"cats": {"sleep": "sixteen hours", "sound": "purr"}}
19docs = splitter.create_documents([data], metadatas=[{"source": "pets.json"}])
20print(len(docs), docs[0].metadata)                      # 3 {'source': 'pets.json'}

↩ back to index