RAG pipeline

Agentoria doesn't ship a vector store — it defines the ports (Retriever, EmbeddingClient) so you plug your own (D1 + Vectorize, pgvector, a search API…). Around them, @agentoria/core provides the pure pipeline pieces that turn a store into good retrieval.

1. Chunk

splitText is a recursive, token-budgeted splitter — it breaks on the largest natural boundary (paragraph → line → sentence → word) that keeps each chunk under budget, then merges back up with overlap so context survives the seams:

import { splitText } from '@agentoria/core';

const chunks = splitText(document, { chunkSize: 512, chunkOverlap: 64 });
// → [{ text, index, tokens }, …]

2. Retrieve — hybrid

Vector search blurs exact terms, names, and codes; BM25 (lexical) catches them. Run both and fuse the rankings with Reciprocal Rank Fusion:

import { createBm25Index, hybridSearch } from '@agentoria/core';

const lexical = createBm25Index(corpus.map((c) => ({ id: c.id, text: c.text })));

const results = await hybridSearch({
  query,
  retriever,   // your vector Retriever
  lexical,     // the BM25 index
  corpus,      // resolves lexical hits back to chunks
  k: 8,
});

RRF is rank-based, so it fuses cosine and BM25 scores without normalizing their different scales.

3. Rerank — diversity

Top-k by score often returns near-duplicates. Maximal Marginal Relevance reranks for relevance and novelty, so the context isn't three copies of the same passage:

import { mmrRerank } from '@agentoria/core';

const diverse = mmrRerank(queryEmbedding, embeddedChunks, { lambda: 0.5, k: 6 });

For a cross-encoder or LLM judge, implement the Reranker port instead.

4. Pack

Fit the best chunks into a token budget — dedup, stop before overflow, and get Citations to render under the answer:

import { packContext } from '@agentoria/core';

const { text, citations } = packContext(diverse, { maxTokens: 2000 });

const system = `Answer using only this context:\n\n${text}`;

As a tool

Wrap a Retriever as an agent tool so the model can search mid-conversation:

import { retrieverTool } from '@agentoria/runtime';

const tools = [retrieverTool(retriever, { defaultK: 6 })];