RAG extract¶

The extract sub-package adds structured metadata to a *rag.Document before chunking: titles, summaries, keywords, suggested questions, named entities, language detection. Extractors enrich a document's Metadata map without changing the content; the chunk stage later copies that metadata onto every chunk, so retrieval and postprocessing can filter/boost based on it (e.g. BM25 field boosts on title).

All implementations satisfy:

type Extractor interface {
    Extract(ctx, doc *rag.Document) (map[string]string, error)
}

The DSL processor rag-extract chains extractors via extract.NewChain(extractors...), and extract.GetExtractor(name, model) dispatches by name. Most extractors are LLM-driven; two are purely statistical (no model needed).

Implementations¶

title — LLM-inferred title¶

rag:
  extract:
    - type: title
      model: ${LLM_REF}

Prompts the LLM with the first 1000 chars of the document, asks for a concise title. Writes title. Use for documents without explicit titles (chat logs, transcripts). Source: pkg/rag/extract/extract.go (TitleExtractor).

summary — LLM-generated summary¶

rag:
  extract:
    - type: summary
      model: ${LLM_REF}
      max_length: 200

Prompts with the first 3000 chars; writes summary. Pairs well with retrieve: summary (small-corpus summary-first retrieval). Source: pkg/rag/extract/extract.go (SummaryExtractor).

keywords — statistical (TF-based) keyword extraction¶

rag:
  extract:
    - type: keywords
      max_count: 10

No LLM. Tokenises lowercase, drops a hand-rolled stop-word list (the, and, is, …), counts term frequency, returns the top max_count. Writes keywords as a comma-separated string.

This is what extract.GetExtractor("keywords", _) returns — fast and deterministic. Source: pkg/rag/extract/extract.go (StatisticalKeywordExtractor).

keywords-llm — LLM-extracted keywords¶

rag:
  extract:
    - type: keywords-llm
      model: ${LLM_REF}

LLM picks salient keywords. Higher quality than keywords for domain-specific content (e.g. medical, legal) where stop-word filtering misses the point. Writes keywords. Source: pkg/rag/extract/extract.go (KeywordsExtractor).

questions — LLM-generated suggested questions¶

rag:
  extract:
    - type: questions
      model: ${LLM_REF}
      max_count: 5

Asks the LLM what questions the document answers. Writes questions_answered (one per line). Pairs naturally with a bm25 retriever whose FieldBoosts includes questions_answered — query phrasing then matches the LLM-generated paraphrases. Source: pkg/rag/extract/extract.go (QuestionsExtractor).

entities — LLM-extracted named entities¶

rag:
  extract:
    - type: entities
      model: ${LLM_REF}

Asks the LLM for named entities (people, organizations, locations, dates) in TYPE:NAME format. Writes entities. Note this is the chunk-metadata-level entity extractor — for a real graph use the graph sub-package's EntityExtractor instead (which writes a typed graph store, not a string). Source: pkg/rag/extract/extract.go (EntityExtractor).

language — heuristic language detection¶

rag:
  extract:
    - type: language

No LLM. Runs seven regex patterns against the document's first 2000 chars and picks the language with the most matches: en, es, fr, de, zh, ja, ko. Returns unknown if no language dominates. Writes language. Use for multilingual corpora where downstream routing depends on language (e.g. picking a multilingual embedder, or filtering by lang=en). Source: pkg/rag/extract/extract.go (LanguageDetector).

Choosing between them¶

A typical extraction chain for production RAG:

rag:
  extract:
    - type: language
    - type: title
      model: ${LLM_REF}
    - type: keywords        # statistical, cheap
    - type: questions
      model: ${LLM_REF}

This costs two LLM calls per document and produces title, keywords, questions, and language — all of which can boost retrieval (e.g. as BM25 field boosts) and feed metadata filters.

See also¶

• ingest — produces the *rag.Document extract enriches.
• chunk — copies extracted metadata onto every chunk.
• retrieve — bm25.FieldBoosts exploits extracted fields (title, keywords, questions_answered).
• graph — for typed entity graphs (instead of string metadata).