RAG Pipeline Ingestion

Why ingestion is the hard part

A retrieval system is only as good as the data you put in it. Retrieval, reranking, and generation get all the attention, but the step that actually breaks production RAG is ingestion — turning a heterogeneous pile of documents into clean, consistently-chunked, embedded records.

Real corpora are messy. You get native PDFs and scanned PDFs, DOCX, PPTX, XLSX, HTML exports, and emails with attachments — each with its own quirks: multi-column layouts, tables that must stay intact, headers and footers that pollute chunks, and reading order that naive extractors scramble. Stitch together five libraries to cover all of that and you've built a fragile pipeline that fails silently on the next unusual file.

What Xberg does

Xberg collapses extraction, layout detection, chunking, and embedding into a single call. Send a file — or a few thousand — and it returns retrieval-ready chunks with embeddings and metadata already attached.

• Extracts text, tables, and structure from 100+ formats — native or scanned.
• Detects layout and reading order, so chunks follow the document, not the raw byte stream.
• Semantic chunking with token-aware sizing, plus keyword extraction and rich metadata.
• Ultra-fast embeddings via a Rust-native ONNX engine — four presets out of the box, extensible to any model.
• Runs in milliseconds per document and is built for batch, so you can index millions without waiting weeks.

In code

Extract, chunk, and embed, then upsert the records straight into your vector store:

from xberg import extract_file
 
# Extract, chunk, and embed in a single call
result = extract_file(
    "q3-report.pdf",
    chunk=True,
    chunk_size=512,
    embed=True,            # Rust-native ONNX embeddings
)
 
# Upsert retrieval-ready records into your vector store
store.upsert([
    {
        "id": c.id,
        "text": c.text,
        "embedding": c.embedding,
        "metadata": c.metadata,
    }
    for c in result.chunks
])

Indexing a whole corpus is the same call over a list — batch processing keeps throughput high:

from xberg import batch_extract
 
# Same call across an entire corpus
results = batch_extract(paths, chunk=True, embed=True)

What you get back

Every chunk comes back with its text, embedding, and metadata — no second pipeline to assemble it:

{
  "metadata": { "title": "Q3 Report", "pages": 12, "format": "pdf" },
  "chunks": [
    {
      "id": "c_0",
      "text": "Revenue grew 24% quarter over quarter…",
      "embedding": [0.013, -0.072, 0.041, …],
      "metadata": { "page": 1, "section": "Summary" }
    }
  ]
}

Why teams pick Xberg for this

• One dependency instead of five — fewer moving parts to break in production.
• Consistent chunking and metadata across every format, so retrieval quality doesn't depend on file type.
• Speed that keeps re-indexing cheap — refresh your knowledge base on a schedule, not a sprint.
• Self-hostable, so sensitive corpora never have to leave your environment.