LearnPracticeExamsMocks

Clear practice, useful explanations and steady progress for exam preparation.

Study

AptitudeLogical reasoningVerbal abilityTechnical aptitudeAI & modern dev

Explore

Practice libraryLocal progressExam patternsFor collegesSearchAbout AISEAFAQ

Trust & support

ContactReport a problemContent correctionsAccessibility
© 2026 AISEA. Built for focused learning.Privacy · Terms · Editorial policy
Home
›AI & Modern Dev
›Text Processing
AI
Prompt Engineering
Vibe Coding
Claude Code
Agent Building
Computer Vision
Text ProcessingPractice

Text processing, tokenization and RAG serving architecture

Explore BPE/WordPiece tokenization, TF-IDF, vector embeddings, chunking strategies, and retrieval-augmented generation serving cost models with worked examples.

100 practice questionsPage updated 24 Sept 2026Review metadata pendingPractice questions
Practise this concept

Text processing is the foundation of modern NLP and AI applications, encompassing everything from basic tokenization to sophisticated RAG pipelines. It bridges raw text data and machine understanding through embedding models, vector databases, and retrieval systems. In 2025, text processing powers search engines, chatbots, document analysis tools, and enterprise knowledge systems.

Key formulas and rules

7 rules
1

TF-IDF Weighting: TF-IDF(t,d,D)=TF(t,d)×log(∣{d∈D:t∈d}∣N​), balancing local term frequency with inverse document frequency across N corpus documents

2

Cosine Similarity: cos(θ)=∥A∥∥B∥A⋅B​=∑i=1n​Ai2​​∑i=1n​Bi2​​∑i=1n​Ai​Bi​​

3

Euclidean Distance (L2): d(A,B)=∑i=1n​(Ai​−Bi​)2​, measuring geometric spatial distance between embedding vectors

4

BM25 Retrieval Function: Score(D,Q)=∑i=1n​IDF(qi​)×f(qi​,D)+k1​⋅(1−b+b⋅avgdl∣D∣​)f(qi​,D)⋅(k1​+1)​

5

Chunk Overlap Fraction: Overlap Ratio=Tokenschunk​Tokensoverlap​​, typically maintained between 10% and 20% in RAG chunking pipelines

6

Recall at Rank K: Recall@K=∣Total Relevant Items∣∣Relevant Items∩Top-K Retrieved∣​

7

Mean Reciprocal Rank (MRR): MRR=∣Q∣1​∑i=1∣Q∣​ranki​1​, where ranki​ is the 1-based position of the first correct hit

Key concepts

Tokenization Strategies

Tokenization converts text into discrete units (tokens) that models process. Modern approaches include: (1) WordPiece/BPE (Byte Pair Encoding) - used by GPT-4, LLaMA, splits words into subword units; (2) SentencePiece - language-agnostic, handles whitespace-free languages; (3) Character-level - used by some specialized models, preserves exact text; (4) Tiktoken - OpenAI's efficient BPE implementation for GPT models. In 2025, tokenization overhead matters: GPT-4o uses ~4 chars/token for English, ~1-2 chars/token for code. Token costs directly impact API pricing.

Embedding Models

Embeddings map text to dense vectors (typically 384-3072 dimensions) capturing semantic meaning. Leading models in 2025: (1) OpenAI text-embedding-3-large (3072d) - highest quality, supports dimension reduction; (2) Cohere embed-v3 - strong multilingual, supports compression; (3) Voyage AI voyage-2 (1024d) - excellent for long documents; (4) Google text-gecko (768d) - integrated with Vertex AI; (5) E5-large-v2 (1024d) - open-source, strong benchmark performance; (6) BGE-large-en - open-source, competitive with commercial models. Model selection depends on language coverage, dimension constraints, cost, and domain specialization.

Vector Databases

Vector databases store and retrieve embeddings efficiently. Top options in 2026: (1) Pinecone - managed, serverless or pod-based, excellent for production; (2) Chroma - lightweight, runs locally, great for development; (3) pgvector - PostgreSQL extension, leverages existing infrastructure; (4) Weaviate - open-source with GraphQL, strong hybrid search; (5) Qdrant - Rust-based, high performance, open-source; (6) Milvus - distributed, handles billions of vectors. Key features to evaluate: metadata filtering, hybrid (vector + keyword) search, multi-tenancy, replication, backup/restore, and pricing model. Pinecone's serverless tier charges per query, while pgvector adds minimal overhead to existing Postgres deployments.

RAG Pipeline Architecture

Retrieval-Augmented Generation (RAG) combines retrieval with generation. Core components: (1) Document Processing - extract text, chunk, embed; (2) Vector Store - persist embeddings with metadata; (3) Query Processing - embed query, retrieve top-K chunks; (4) Context Assembly - format retrieved chunks with query; (5) Generation - LLM synthesizes answer from context. Advanced patterns: Hybrid RAG (vector + BM25), Reranking (Cross-encoder after retrieval), Query Expansion (generate multiple query variations), Self-Querying (LLM generates metadata filters), Multi-Vector Store (separate indexes by document type). In 2026, production RAG systems use 100-500 token chunks with 10-20% overlap and retrieve 5-20 documents per query.

Chunking Strategies

Chunking splits documents into retrievable units. Common approaches: (1) Fixed-size - simple, predictable token counts, risks mid-sentence splits; (2) Recursive character - splits on paragraphs, then sentences, then words; (3) Structural splitting - respects document hierarchy (markdown headings, section tags, legal clauses); (4) Semantic chunking - computes embedding distances between consecutive sentences to detect topic shifts; (5) Sentence-based - NLTK/spaCy sentence boundaries; (6) Sliding window - overlapping chunks for context continuity; (7) Parent-child - retrieve small chunks, return larger parent context. Best practices: 256-512 tokens for dense text, 512-1024 for code/technical docs, 10-20% overlap preserves context across boundaries.

Named Entity Recognition (NER)

NER identifies and classifies entities in text (people, organizations, locations, dates, etc.). Modern approaches: (1) Transformer-based - spaCy's transformer pipeline, Hugging Face models (xlm-roberta-ner); (2) LLM extraction - prompt GPT-4/Claude with structured output; (3) Fine-tuned models - domain-specific (biomedical, legal, financial). In 2026, spaCy's en_core_web_trf achieves ~92% F1 on standard benchmarks. For production: define entity schema upfront, handle entity disambiguation (Apple the company vs. fruit), validate extracted entities, and handle edge cases (multi-word entities, nested entities). LLM-based NER offers flexibility but higher latency and cost.

Sentiment Analysis

Sentiment analysis classifies text emotional tone. Approaches: (1) Lexicon-based - VADER, AFINN, fast but limited nuance; (2) ML classifiers - Naive Bayes, SVM, trained on labeled data; (3) Deep learning - BERT-based models (cardiffnlp/twitter-roberta-base-sentiment); (4) LLM prompting - GPT-4/Claude with structured output. Modern production systems often combine: fast lexicon for real-time, transformer for batch, LLM for ambiguous cases. Output formats: binary (positive/negative), ternary (pos/neg/neutral), fine-grained (5-star), aspect-based (sentiment per feature). Key challenge: sarcasm, negation, and domain-specific language require contextual understanding.

Text Classification

Text classification assigns predefined categories to text. Methods: (1) Traditional ML - TF-IDF + SVM/Naive Bayes, still effective for simple tasks; (2) Fine-tuned transformers - BERT/RoBERTa with classification head; (3) Zero-shot - classify without training data using NLI models or LLMs; (4) Few-shot - LLM with example prompts. In 2026, zero-shot classification via CLIP-style models or LLM prompting handles many use cases without training data. For high-volume production: fine-tune smaller models (DistilBERT, MiniLM) for speed. Evaluate using precision, recall, F1 per class - macro-averaged for balanced class importance, weighted for imbalanced datasets.

Check the idea while it is fresh.Open a focused exercise without losing this article.Practise this concept

Worked examples

Example 1

Calculate the TF-IDF score for the term 'machine' in a document where 'machine' appears 5 times, the document has 100 total words, and the term appears in 50 documents out of a corpus of 10,000 documents.

Solution

Step 1: Calculate Term Frequency (TF)
TF = count(term in doc) / total terms in doc
TF=5/100=0.05

Step 2: Calculate Inverse Document Frequency (IDF) using base-10 logarithm
IDF = \log_{10}(N / df(term))
IDF = \log_{10}(10000 / 50) = \log_{10}(200) ≈ 2.301

Step 3: Calculate TF-IDF
TF−IDF=TF × IDF
TF−IDF=0.05 × 2.301 = 0.115

Answer: The TF-IDF score for 'machine' is approximately 0.115. This indicates strong specificity: 'machine' appears frequently in this specific document (TF=0.05) and is highly selective across the corpus, appearing in only 0.5% (50/10,000) of all documents.

Example 2

A RAG system retrieves 10 chunks from a vector database. The query embedding has cosine similarity scores: [0.89, 0.85, 0.82, 0.78, 0.75, 0.72, 0.68, 0.65, 0.61, 0.58]. Calculate the Mean Reciprocal Rank (MRR) if the relevant answer appears in chunks at positions 1, 3, and 7. Then explain why MRR matters for RAG evaluation.

Solution

Step 1: Identify the first relevant result
The first relevant chunk is at position 1 (0-indexed: position 0)
Reciprocal rank = 1/(position + 1) = 1/(0 + 1) = 1

Step 2: Calculate MRR for this single query
MRR = 1 / (first relevant position + 1)
MRR=1/1=1.0

Step 3: Interpret the result
MRR=1.0 is the best possible score - the top result was relevant.

Step 4: Why MRR matters for RAG
MRR measures how early relevant results appear, not just whether they appear. For RAG, this is critical because: (1) LLMs have limited context windows - early results get more attention; (2) Users prefer concise answers - finding relevance quickly reduces context bloat; (3) Reranking models can improve MRR by promoting relevant chunks.

Answer: MRR=1.0 for this query. For multi-query evaluation, average the reciprocal ranks across all queries. A production RAG system should target MRR > 0.7 for top-10 retrieval.

Example 3

Design a chunking strategy for a legal document repository containing 500 PDFs averaging 50 pages each. Documents contain sections, subsections, and clause numbering. The goal is RAG for legal Q&A. Specify chunk size, overlap, splitting strategy, and justify each choice.

Solution

Step 1: Analyze document characteristics
- Legal documents have hierarchical structure (sections, subsections, clauses)
- Cross-references between sections are common
- Each clause may be 100-500 tokens
- Context window target: 8000 tokens for generation

Step 2: Choose splitting strategy
Use structural and semantic chunking respecting document hierarchy:
- Primary separator: section headings (regex match 'Section \d+')
- Secondary separator: subsection headings
- Tertiary: clause boundaries (numbered lists like '1.1', '1.2')

Step 3: Determine chunk parameters
- Target chunk size: 512 tokens (covers typical legal clause + context)
- Overlap: 100 tokens (20%) - captures cross-references
- Maximum chunk: 1024 tokens - merge short adjacent clauses
- Minimum chunk: 128 tokens - avoid over-fragmentation

Step 4: Add metadata for filtering
Store with each chunk:
- Document title, date, jurisdiction
- Section/subsection path
- Clause numbers referenced

Step 5: Justification
- 512 tokens balances retrieval precision (smaller = more specific) with context preservation
- 20% overlap captures references that span clause boundaries
- Structural splitting respects legal document hierarchy
- Metadata enables jurisdiction-based filtering before vector search

Answer: Structural splitting with 512-token target, 100-token overlap on section/clause boundaries, paired with legal metadata (jurisdiction, document type, date) for pre-filtering.

Example 4

Compare embedding models for a multilingual RAG system serving English, Spanish, and Mandarin documents. The system processes 1M queries/month with average 5 chunks retrieved per query. Compare: OpenAI text-embedding-3-large, Cohere embed-multilingual-v3, and a self-hosted E5 model on dimensions, serving cost, latency, and quality.

Solution

Step 1: Define production RAG cost and latency model
- Document chunk embeddings are precomputed once during ingestion, so serving-time embedding costs depend strictly on incoming queries.
- Average query length: ~30 tokens. For 1M queries/month, total query tokens to embed = 30M tokens/month.
- Latency: query embedding time + vector retrieval + generation.

Step 2: Analyze each embedding model

OpenAI text-embedding-3-large:
- Dimensions: 3072 (supports Matryoshka reduction down to 256/512/1024)
- Serving Cost: $0.13 per 1M tokens. For 30M query tokens = 30 × $0.13 = $3.90/month
- Latency: ~50-90ms via API
- Quality: Exceptional English benchmark scores, very good multilingual retrieval

Cohere embed-multilingual-v3:
- Dimensions: 1024
- Serving Cost: $0.10 per 1M tokens. For 30M query tokens = 30 × $0.10 = $3.00/month
- Latency: ~30-60ms via API
- Quality: State-of-the-art multilingual performance (specifically tuned across 100+ languages)

Self-hosted E5-mistral-7b-instruct:
- Dimensions: 4096
- Serving Cost: Requires dedicated GPU compute (e.g. 1× A10G/L4 instance at ~$0.70-1.00/hour × 720 hours = ~$500-720/month)
- Latency: ~10-25ms (local VPC, no external internet round-trip)
- Quality: Top-tier retrieval, but requires ongoing model maintenance, auto-scaling, and health monitoring

Step 3: Recommendation
For a multilingual English/Spanish/Mandarin system at 1M queries/month:
- Cohere embed-multilingual-v3 is optimal: superior native multilingual accuracy, minimal serving cost (~$3.00/month), and zero infrastructure maintenance overhead.
- OpenAI text-embedding-3-large is an excellent alternative (~$3.90/month) with flexible Matryoshka dimension truncation.
- Self-hosted GPU infrastructure costs ~$500/month or more, which is over 100× more expensive for 1M queries/month, making hosted APIs drastically more economical until query volume exceeds ~100M+ queries/month.

Answer: Cohere embed-multilingual-v3 at ~$3.00/month serving cost, 1024 dimensions, 30-60ms API latency, best-in-class multilingual quality.

Representative solved questions

See the kind of question in this topic before opening the full practice set.

Question 1

What is the Byte-Pair Encoding (BPE) tokenization algorithm used for?

  1. A.

    Encoding text as raw bytes

  2. B.

    Encrypting sensitive text data

  3. C.

    Compressing files for storage

  4. D.

    Creating subword vocabularies by iteratively merging frequent character pairs

Answer: D. Creating subword vocabularies by iteratively merging frequent character pairs

Explanation

Step 1: BPE starts with a character-level vocabulary and iteratively merges frequent pairs.
Step 2: This creates a subword vocabulary that balances vocabulary size with sequence length.
Step 3: Models like GPT-2 and GPT-3 use BPE tokenization to handle diverse vocabulary efficiently.
Answer: Creating subword vocabularies by merging frequent character pairs.

Sources and review notes

This is an AISEA-authored practice question.

Review status: accepted · Reviewed 2026-08-13 · structure and answer-key checks, editorial quality checks, duplicate screening

Question 2

What is the difference between Word2Vec's Skip-gram and CBOW architectures?

  1. A.

    Skip-gram is for classification; CBOW is for generation

  2. B.

    Skip-gram is faster; CBOW is more accurate

  3. C.

    Skip-gram predicts context from target word; CBOW predicts target from context

  4. D.

    Skip-gram uses character n-grams; CBOW uses word n-grams

Answer: C. Skip-gram predicts context from target word; CBOW predicts target from context

Explanation

Step 1: Skip-gram takes a target word and predicts surrounding context words.
Step 2: CBOW (Continuous Bag of Words) takes context words and predicts the target word.
Step 3: Skip-gram works better for rare words; CBOW is faster and smoother for frequent words.
Answer: Skip-gram predicts context from target; CBOW predicts target from context.

Sources and review notes

This is an AISEA-authored practice question.

Review status: accepted · Reviewed 2026-08-13 · structure and answer-key checks, editorial quality checks, duplicate screening

Question 3

What is the formula for TF-IDF score?

  1. A.

    TF−IDF=TF/IDF

  2. B.

    TF−IDF=TF+IDF

  3. C.

    TF−IDF=TF × IDF = (Term count / Total terms) × log(Total docs / Docs with term)

  4. D.

    TF−IDF=log(TF) × log(IDF)

Answer: C. TF-IDF = TF × IDF = (Term count / Total terms) × log(Total docs / Docs with term)

Explanation

Step 1: Term Frequency (TF) = number of times term appears in document / total terms.
Step 2: Inverse Document Frequency (IDF) = log(total documents / documents containing the term).
Step 3: The TF-IDF score is the product: higher scores indicate important, distinctive terms.
Answer: TF−IDF=TF × IDF with the given formula.

Sources and review notes

This is an AISEA-authored practice question.

Review status: accepted · Reviewed 2026-08-13 · structure and answer-key checks, editorial quality checks, duplicate screening

Common mistakes and useful habits

  • Start with off-the-shelf embedding models (text-embedding-3-small, Cohere embed) before fine-tuning. Fine-tuning embeddings requires 10K+ labeled query-document pairs and often provides marginal gains.
  • Use dimension reduction (Matryoshka embeddings) to cut storage 4x with minimal quality loss. OpenAI text-embedding-3-large supports 256, 512, 1024, or 3072 dimensions.
  • Always evaluate retrieval with domain-specific test queries. Generic benchmarks (MTEB) don't reflect your actual use case. Build a test set of 100+ real queries with ground-truth relevant documents.
  • Hybrid search (vector + BM25) typically outperforms pure vector search by 5-15% on recall. Use vector for semantic matching, BM25 for exact term matching. Most vector databases (Pinecone, Weaviate, pgvector) support hybrid.
  • Chunk overlap is your friend for context preservation. 10-20% overlap prevents information loss at chunk boundaries. For technical documents, consider 25% overlap.
  • Pre-filter by metadata before vector search when possible. Filter on date ranges, document types, or categories to reduce search space and improve both speed and precision.

Ready to test your understanding?

Work through 100 questions with explanations after each answer.

Start practice
Next topic: Prompt Engineering →

Keep building the same skill family.

Prompt Engineering

Design effective prompts for LLMs

Open practice
Vibe Coding

AI-assisted coding workflows and techniques

Open practice
Claude Code

Using Claude Code for agentic development

Open practice
On this page
Formulas and rulesKey conceptsWorked examplesRepresentative questionsCommon mistakesRelated practice

Suggest a correction