Text Processing

Tokenization converts text into discrete units (tokens) that models process. Modern approaches include: (1) WordPiece/BPE (Byte Pair Encoding) - used by GPT-4, LLaMA, splits words into subword units; (2) SentencePiece - language-agnostic, handles whitespace-free languages; (3) Character-level - used by some specialized models, preserves exact text; (4) Tiktoken - OpenAI's efficient BPE implementation for GPT models. In 2025, tokenization overhead matters: GPT-4o uses ~4 chars/token for English, ~1-2 chars/token for code. Token costs directly impact API pricing.

Embeddings map text to dense vectors (typically 384-3072 dimensions) capturing semantic meaning. Leading models in 2025: (1) OpenAI text-embedding-3-large (3072d) - highest quality, supports dimension reduction; (2) Cohere embed-v3 - strong multilingual, supports compression; (3) Voyage AI voyage-2 (1024d) - excellent for long documents; (4) Google text-gecko (768d) - integrated with Vertex AI; (5) E5-large-v2 (1024d) - open-source, strong benchmark performance; (6) BGE-large-en - open-source, competitive with commercial models. Model selection depends on language coverage, dimension constraints, cost, and domain specialization.

Vector databases store and retrieve embeddings efficiently. Top options in 2025: (1) Pinecone - managed, serverless or pod-based, excellent for production; (2) Chroma - lightweight, runs locally, great for development; (3) pgvector - PostgreSQL extension, leverages existing infrastructure; (4) Weaviate - open-source with GraphQL, strong hybrid search; (5) Qdrant - Rust-based, high performance, open-source; (6) Milvus - distributed, handles billions of vectors. Key features to evaluate: metadata filtering, hybrid (vector + keyword) search, multi-tenancy, replication, backup/restore, and pricing model. Pinecone's serverless tier charges per query, while pgvector adds minimal overhead to existing Postgres deployments.

Retrieval-Augmented Generation (RAG) combines retrieval with generation. Core components: (1) Document Processing - extract text, chunk, embed; (2) Vector Store - persist embeddings with metadata; (3) Query Processing - embed query, retrieve top-K chunks; (4) Context Assembly - format retrieved chunks with query; (5) Generation - LLM synthesizes answer from context. Advanced patterns: Hybrid RAG (vector + BM25), Reranking (Cross-encoder after retrieval), Query Expansion (generate multiple query variations), Self-Querying (LLM generates metadata filters), Multi-Vector Store (separate indexes by document type). In 2025, production RAG systems use 100-500 token chunks with 10-20% overlap and retrieve 5-20 documents per query.

Chunking splits documents into retrievable units. Common approaches: (1) Fixed-size - simple, predictable token counts, risks mid-sentence splits; (2) Recursive character - splits on paragraphs, then sentences, then words; (3) Semantic - respects document structure (headings, sections); (4) Sentence-based - NLTK/spaCy sentence boundaries; (5) Sliding window - overlapping chunks for context continuity; (6) Parent-child - retrieve small chunks, return larger parent context. Best practices: 256-512 tokens for dense text, 512-1024 for code/technical docs, 10-20% overlap preserves context across boundaries. LangChain's RecursiveCharacterTextSplitter with separators ['\n\n', '\n', '. ', ' ', ''] handles most documents well.

NER identifies and classifies entities in text (people, organizations, locations, dates, etc.). Modern approaches: (1) Transformer-based - spaCy's transformer pipeline, Hugging Face models (xlm-roberta-ner); (2) LLM extraction - prompt GPT-4/Claude with structured output; (3) Fine-tuned models - domain-specific (biomedical, legal, financial). In 2025, spaCy's en_core_web_trf achieves ~92% F1 on standard benchmarks. For production: define entity schema upfront, handle entity disambiguation (Apple the company vs. fruit), validate extracted entities, and handle edge cases (multi-word entities, nested entities). LLM-based NER offers flexibility but higher latency and cost.

Sentiment analysis classifies text emotional tone. Approaches: (1) Lexicon-based - VADER, AFINN, fast but limited nuance; (2) ML classifiers - Naive Bayes, SVM, trained on labeled data; (3) Deep learning - BERT-based models (cardiffnlp/twitter-roberta-base-sentiment); (4) LLM prompting - GPT-4/Claude with structured output. Modern production systems often combine: fast lexicon for real-time, transformer for batch, LLM for ambiguous cases. Output formats: binary (positive/negative), ternary (pos/neg/neutral), fine-grained (5-star), aspect-based (sentiment per feature). Key challenge: sarcasm, negation, and domain-specific language require contextual understanding.

Text classification assigns predefined categories to text. Methods: (1) Traditional ML - TF-IDF + SVM/Naive Bayes, still effective for simple tasks; (2) Fine-tuned transformers - BERT/RoBERTa with classification head; (3) Zero-shot - classify without training data using NLI models or LLMs; (4) Few-shot - LLM with example prompts. In 2025, zero-shot classification via CLIP-style models or LLM prompting handles many use cases without training data. For high-volume production: fine-tune smaller models (DistilBERT, MiniLM) for speed. Evaluate using precision, recall, F1 per class - macro-averaged for balanced class importance, weighted for imbalanced datasets.