Text Processing
NLP, embeddings, classification, RAG
Text processing is the foundation of modern NLP and AI applications, encompassing everything from basic tokenization to sophisticated RAG pipelines. It bridges raw text data and machine understanding through embedding models, vector databases, and retrieval systems. In 2025, text processing powers search engines, chatbots, document analysis tools, and enterprise knowledge systems.
Key formulas and rules
Key concepts
Tokenization Strategies
Tokenization converts text into discrete units (tokens) that models process. Modern approaches include: (1) WordPiece/BPE (Byte Pair Encoding) - used by GPT-4, LLaMA, splits words into subword units; (2) SentencePiece - language-agnostic, handles whitespace-free languages; (3) Character-level - used by some specialized models, preserves exact text; (4) Tiktoken - OpenAI's efficient BPE implementation for GPT models. In 2025, tokenization overhead matters: GPT-4o uses ~4 chars/token for English, ~1-2 chars/token for code. Token costs directly impact API pricing.
Embedding Models
Embeddings map text to dense vectors (typically 384-3072 dimensions) capturing semantic meaning. Leading models in 2025: (1) OpenAI text-embedding-3-large (3072d) - highest quality, supports dimension reduction; (2) Cohere embed-v3 - strong multilingual, supports compression; (3) Voyage AI voyage-2 (1024d) - excellent for long documents; (4) Google text-gecko (768d) - integrated with Vertex AI; (5) E5-large-v2 (1024d) - open-source, strong benchmark performance; (6) BGE-large-en - open-source, competitive with commercial models. Model selection depends on language coverage, dimension constraints, cost, and domain specialization.
Vector Databases
Vector databases store and retrieve embeddings efficiently. Top options in 2025: (1) Pinecone - managed, serverless or pod-based, excellent for production; (2) Chroma - lightweight, runs locally, great for development; (3) pgvector - PostgreSQL extension, leverages existing infrastructure; (4) Weaviate - open-source with GraphQL, strong hybrid search; (5) Qdrant - Rust-based, high performance, open-source; (6) Milvus - distributed, handles billions of vectors. Key features to evaluate: metadata filtering, hybrid (vector + keyword) search, multi-tenancy, replication, backup/restore, and pricing model. Pinecone's serverless tier charges per query, while pgvector adds minimal overhead to existing Postgres deployments.
RAG Pipeline Architecture
Retrieval-Augmented Generation (RAG) combines retrieval with generation. Core components: (1) Document Processing - extract text, chunk, embed; (2) Vector Store - persist embeddings with metadata; (3) Query Processing - embed query, retrieve top-K chunks; (4) Context Assembly - format retrieved chunks with query; (5) Generation - LLM synthesizes answer from context. Advanced patterns: Hybrid RAG (vector + BM25), Reranking (Cross-encoder after retrieval), Query Expansion (generate multiple query variations), Self-Querying (LLM generates metadata filters), Multi-Vector Store (separate indexes by document type). In 2025, production RAG systems use 100-500 token chunks with 10-20% overlap and retrieve 5-20 documents per query.
Chunking Strategies
Chunking splits documents into retrievable units. Common approaches: (1) Fixed-size - simple, predictable token counts, risks mid-sentence splits; (2) Recursive character - splits on paragraphs, then sentences, then words; (3) Semantic - respects document structure (headings, sections); (4) Sentence-based - NLTK/spaCy sentence boundaries; (5) Sliding window - overlapping chunks for context continuity; (6) Parent-child - retrieve small chunks, return larger parent context. Best practices: 256-512 tokens for dense text, 512-1024 for code/technical docs, 10-20% overlap preserves context across boundaries. LangChain's RecursiveCharacterTextSplitter with separators ['\n\n', '\n', '. ', ' ', ''] handles most documents well.
Worked examples
Example 1
Calculate the TF-IDF score for the term 'machine' in a document where 'machine' appears 5 times, the document has 100 total words, and the term appears in 50 documents out of a corpus of 10,000 documents.
Step 1: Calculate Term Frequency (TF)
TF = count(term in doc) / total terms in doc
TF = 5 / 100 = 0.05
Step 2: Calculate Inverse Document Frequency (IDF)
IDF = log(N / df(term))
IDF = log(10000 / 50) = log(200) ≈ 2.301
Step 3: Calculate TF-IDF
TF-IDF = TF × IDF
TF-IDF = 0.05 × 2.301 = 0.115
Answer: The TF-IDF score for 'machine' is approximately 0.115. This indicates moderate importance - the term appears somewhat frequently in this document but is also present in many other documents.
Example 2
A RAG system retrieves 10 chunks from a vector database. The query embedding has cosine similarity scores: [0.89, 0.85, 0.82, 0.78, 0.75, 0.72, 0.68, 0.65, 0.61, 0.58]. Calculate the Mean Reciprocal Rank (MRR) if the relevant answer appears in chunks at positions 1, 3, and 7. Then explain why MRR matters for RAG evaluation.
Step 1: Identify the first relevant result
The first relevant chunk is at position 1 (0-indexed: position 0)
Reciprocal rank = 1/(position + 1) = 1/(0 + 1) = 1
Step 2: Calculate MRR for this single query
MRR = 1 / (first relevant position + 1)
MRR = 1 / 1 = 1.0
Step 3: Interpret the result
MRR = 1.0 is the best possible score - the top result was relevant.
Step 4: Why MRR matters for RAG
MRR measures how early relevant results appear, not just whether they appear. For RAG, this is critical because: (1) LLMs have limited context windows - early results get more attention; (2) Users prefer concise answers - finding relevance quickly reduces context bloat; (3) Reranking models can improve MRR by promoting relevant chunks.
Answer: MRR = 1.0 for this query. For multi-query evaluation, average the reciprocal ranks across all queries. A production RAG system should target MRR > 0.7 for top-10 retrieval.
Example 3
Design a chunking strategy for a legal document repository containing 500 PDFs averaging 50 pages each. Documents contain sections, subsections, and clause numbering. The goal is RAG for legal Q&A. Specify chunk size, overlap, splitting strategy, and justify each choice.
Step 1: Analyze document characteristics
- Legal documents have hierarchical structure (sections, subsections, clauses)
- Cross-references between sections are common
- Each clause may be 100-500 tokens
- Context window target: 8000 tokens for generation
Step 2: Choose splitting strategy
Use semantic chunking respecting document structure:
- Primary separator: section headings (regex match 'Section \d+')
- Secondary separator: subsection headings
- Tertiary: clause boundaries (numbered lists like '1.1', '1.2')
Step 3: Determine chunk parameters
- Target chunk size: 512 tokens (covers typical legal clause + context)
- Overlap: 100 tokens (20%) - captures cross-references
- Maximum chunk: 1024 tokens - merge short adjacent clauses
- Minimum chunk: 128 tokens - avoid over-fragmentation
Step 4: Add metadata for filtering
Store with each chunk:
- Document title, date, jurisdiction
- Section/subsection path
- Clause numbers referenced
Step 5: Justification
- 512 tokens balances retrieval precision (smaller = more specific) with context preservation
- 20% overlap captures references that span clause boundaries
- Semantic splitting respects legal document structure
- Metadata enables jurisdiction-based filtering before vector search
Answer: Semantic chunking with 512-token target, 100-token overlap, splitting on section/clause boundaries, with legal metadata (jurisdiction, document type, date) for pre-filtering.
Representative solved questions
See the kind of question in this topic before opening the full practice set.
Question 1
What is the primary purpose of tokenization in NLP pipelines?
Translating text between languages
Encrypting text for secure transmission
Breaking text into smaller units (tokens) for model processing
Compressing text to reduce storage size
Answer: C. Breaking text into smaller units (tokens) for model processing
ExplanationStep 1: Tokenization is the process of breaking down text into smaller units called tokens.
Step 2: These tokens can be words, subwords, or characters depending on the tokenizer.
Step 3: Models process tokens as input IDs, making tokenization essential for all NLP tasks.
Answer: Breaking text into smaller units for model processing.
Sources and review notes
This is an AISEA-authored practice question.
Review status: accepted · Reviewed 2026-08-13 · structure and answer-key checks, editorial quality checks, duplicate screening
Question 2
Which tokenizer uses subword units and is commonly used with BERT models?
Character-level tokenizer
Whitespace tokenizer
Sentence tokenizer
WordPiece tokenizer
Answer: D. WordPiece tokenizer
ExplanationStep 1: BERT uses the WordPiece tokenization algorithm.
Step 2: WordPiece breaks words into subword units, allowing handling of unknown words.
Step 3: For example, 'unhappiness' might become ['un', '##happiness'] or similar subwords.
Answer: WordPiece tokenizer.
Sources and review notes
This is an AISEA-authored practice question.
Review status: accepted · Reviewed 2026-08-13 · structure and answer-key checks, editorial quality checks, duplicate screening
Question 3
What does the '##' prefix indicate in BERT's WordPiece tokenization?
The token is a special classification token
The token should be ignored during processing
Common mistakes and useful habits
- Start with off-the-shelf embedding models (text-embedding-3-small, Cohere embed) before fine-tuning. Fine-tuning embeddings requires 10K+ labeled query-document pairs and often provides marginal gains.
- Use dimension reduction (Matryoshka embeddings) to cut storage 4x with minimal quality loss. OpenAI text-embedding-3-large supports 256, 512, 1024, or 3072 dimensions.
- Always evaluate retrieval with domain-specific test queries. Generic benchmarks (MTEB) don't reflect your actual use case. Build a test set of 100+ real queries with ground-truth relevant documents.
- Hybrid search (vector + BM25) typically outperforms pure vector search by 5-15% on recall. Use vector for semantic matching, BM25 for exact term matching. Most vector databases (Pinecone, Weaviate, pgvector) support hybrid.
- Chunk overlap is your friend for context preservation. 10-20% overlap prevents information loss at chunk boundaries. For technical documents, consider 25% overlap.
- Pre-filter by metadata before vector search when possible. Filter on date ranges, document types, or categories to reduce search space and improve both speed and precision.
Ready to test your understanding?
Work through 100 questions with explanations after each answer.