Skip to main contentSkip to footer
ExamplescriptintermediateRunnableresearch-brief

Chunking

Runnable example (intermediate) for script using docling, ipykernel.

Key Facts

Level
intermediate
Runtime
Python • OpenAI API
Pattern
Context-backed research with explicit evidence
Interaction
Live sandbox • Script
Updated
14 March 2026

Navigate this example

Linked principles

Apply progressive disclosure to system agencyExpose meaningful operational state, not internal complexityEstablish trust through inspectability

High-level flow

How this example moves from input to execution and reviewable output
Chunking -> User request -> System execution -> Reviewable output -> Apply progressive disclosure to… -> Expose meaningful operational state,…

Trigger

Chunking

Runtime

User request

Outcome

System execution

Why this page exists

This example is shown as both real source code and a product-facing interaction pattern so learners can connect implementation, UX, and doctrine without leaving the library.

Visual flowReal sourceSandbox or walkthroughMCP access
How should this example be used in the platform?

Use the sandbox to understand the experience pattern first, then inspect the source to see how the product boundary, model boundary, and doctrine boundary are actually implemented.

UX pattern: Context-backed research with explicit evidence
Apply progressive disclosure to system agency
Expose meaningful operational state, not internal complexity
Establish trust through inspectability
Source references
Library entry
knowledge-docling-2-chunking
Source path
content/example-library/sources/knowledge/docling/2-chunking.py
Libraries
docling, ipykernel, lancedb, openai, pydantic, python-dotenv, requests, streamlit, tiktoken
Runtime requirements
OPENAI_API_KEY, streamlit
Related principles
Apply progressive disclosure to system agency, Expose meaningful operational state, not internal complexity, Establish trust through inspectability

2-chunking.py

python
from docling.chunking import HybridChunker
from docling.document_converter import DocumentConverter
from dotenv import load_dotenv
from openai import OpenAI
from utils.tokenizer import OpenAITokenizerWrapper

load_dotenv()

# Initialize OpenAI client (make sure you have OPENAI_API_KEY in your environment variables)
client = OpenAI()


tokenizer = OpenAITokenizerWrapper()  # Load our custom tokenizer for OpenAI
MAX_TOKENS = 8191  # text-embedding-3-large's maximum context length


# --------------------------------------------------------------
# Extract the data
# --------------------------------------------------------------

converter = DocumentConverter()
result = converter.convert("https://arxiv.org/pdf/2408.09869")


# --------------------------------------------------------------
# Apply hybrid chunking
# --------------------------------------------------------------

chunker = HybridChunker(
    tokenizer=tokenizer,
    max_tokens=MAX_TOKENS,
    merge_peers=True,
)

chunk_iter = chunker.chunk(dl_doc=result.document)
chunks = list(chunk_iter)

len(chunks)
What should the learner inspect in the code?

Look for the exact place where system scope is bounded: schema definitions, prompt framing, runtime configuration, and the call site that turns user intent into a concrete model or workflow action.

Look for output contracts and validation
Look for the exact execution call
Look for what the product could expose to the user
How does the sandbox relate to the source?

The sandbox should make the UX legible: what the user sees, what the system is deciding, and how the result becomes reviewable. The source then shows how that behavior is actually implemented.

Enter a question or load a sample query.
Run the search or retrieval step.
Review the final brief with sources and retrieved context.
SandboxContext-backed research with explicit evidence
Research brief lab

This sandbox shows how a search or retrieval request should expose query planning, retrieved context, and the final answer.

UX explanation

The user should not only see a final answer. The product should reveal what was searched, what context shaped the response, and where the system boundary stops.

AI design explanation

These examples combine retrieval, web search, or file context with a synthesis step. The best surface exposes search plan, useful evidence, and a reviewable output.

Interaction walkthrough

  1. 1Enter a question or load a sample query.
  2. 2Run the search or retrieval step.
  3. 3Review the final brief with sources and retrieved context.

Principles in focus

Apply progressive disclosure to system agencyExpose meaningful operational state, not internal complexityEstablish trust through inspectability

Research question

Search planRetrieved context

Plan

The search plan appears here.

Final brief

The final brief appears here alongside the context used.

Used in courses and paths

This example currently stands on its own in the library, but it still connects to the principle system and the broader example family.

Related implementation patterns

Related principles

Runtime architecture

Use this example in your agents

This example is also available through the blueprint’s agent-ready layer. Use the For agents page for the public MCP, deterministic exports, and Claude/Cursor setup.

Define triggers, context, and boundaries before increasing autonomy
Make control, observability, and recovery explicit in the runtime
Choose the right operational patterns before delegating to workflows
Open For agents