Computer vision, CNNs, ViTs and multimodal models
Study convolutional networks, vision transformers, NMS-free object detectors (YOLOv10, RT-DETR), and frontier multimodal vision-language models with worked problems.
Computer Vision (CV) enables machines to interpret and understand visual information from the world. It spans classical techniques like edge detection and feature extraction to modern deep learning approaches using Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). Today's multimodal frontier models like GPT-4o, Claude 3.5/3.7 Sonnet, and Gemini 1.5/2.0 Flash can analyze high-resolution images alongside structured text, opening unprecedented capabilities for spatial reasoning and multimodal understanding.
Key formulas and rules
7 rulesIntersection over Union (IoU):
Convolutional Output Dimension: , where is input size, is kernel size, is padding, and is stride
Softmax Activation: transforms logits into a normalized probability distribution
Categorical Cross-Entropy Loss: , penalizing divergence between ground truth and predicted probability
Precision, Recall & F1:
Feature Map Output Tensor: where spatial dimensions follow the convolution formula and equals the number of filter channels
Mean Average Precision (mAP): , averaging the area under the precision-recall curve across all object classes
Key concepts
Convolutional Neural Networks (CNNs)
CNNs are the foundational architecture for image processing. They use learnable filters (kernels) that slide across input images to extract hierarchical features-from edges and textures in early layers to complex objects in deeper layers. Key components include convolutional layers (feature extraction), pooling layers (spatial downsampling), batch normalization (training stability), and fully connected layers (classification). Popular architectures include ResNet (residual connections), EfficientNet (compound scaling), and MobileNet (depthwise separable convolutions for efficiency).
Object Detection Architectures
Object detection localizes and classifies multiple objects in an image. Two-stage detectors like Faster R-CNN first propose regions of interest, then classify them-achieving higher accuracy at the cost of lower frame rates. Single-stage detectors like YOLO (You Only Look Once) and SSD predict bounding boxes and classes in one forward pass. Modern advancements include NMS-free architectures: YOLOv10 incorporates consistent dual assignments for NMS-free training, and RT-DETR utilizes transformer Hungarian bipartite matching to generate predictions directly without heuristic post-processing latency.
Image Segmentation
Segmentation assigns a class to each pixel. Semantic segmentation treats all instances of a class identically (e.g., all 'car' pixels merged), using architectures like U-Net, DeepLabv3+, and SegFormer. Instance segmentation distinguishes individual objects, combining detection with pixel masks-Mask R-CNN is the classic approach, with newer models like YOLOv8-seg and Mask2Former. Panoptic segmentation unifies both, assigning each pixel to either a semantic class or a specific instance identifier.
Vision Transformers (ViT)
Vision Transformers apply the transformer architecture to images by splitting them into patches, linearly embedding each patch, adding position embeddings, and processing through standard transformer encoder blocks. ViTs excel with large-scale pretraining (JFT-300M, ImageNet-21K) and can outperform CNNs on large datasets. Variants include DeiT (data-efficient training), Swin Transformer (hierarchical with shifted windows), and ViT-22B (largest vision model). ViTs lack inductive bias for translation invariance, requiring more data but offering better scalability and global context modeling.
Transfer Learning and Fine-tuning
Transfer learning leverages pretrained models on large datasets (ImageNet, LAION) for downstream tasks with limited data. Common approaches include feature extraction (freeze backbone, train classifier head), fine-tuning (unfreeze and train with lower learning rate), and progressive unfreezing. Pretrained weights are available via torchvision, timm (PyTorch Image Models), Hugging Face, and TensorFlow Hub. For vision tasks, even small datasets (100-1000 images) can achieve strong results with proper transfer learning and data augmentation.
Multimodal Vision-Language Models
Modern multimodal models combine vision and language understanding natively. Models like GPT-4o, Claude 3.5/3.7 Sonnet, and Gemini 1.5/2.0 Flash accept image and video inputs to perform fine-grained visual reasoning, OCR, chart interpretation, and spatial visual question answering. Open-weights models like Qwen2-VL, LLaVA-NeXT, and PaliGemma align high-resolution vision encoders (SigLIP, ViT) with decoder-only language backbones via cross-attention or projection layers. These systems unify NLP and CV into a single multimodal token space.
Feature Extraction and Embeddings
Visual feature extraction converts images into dense vector representations. Classical methods (SIFT, HOG, SURF) extract hand-crafted features based on gradients and keypoints. Deep learning approaches use pretrained CNN backbones (ResNet, EfficientNet) or vision encoders (CLIP, DINO, DINOv2) to produce embeddings-typically 512-2048 dimensional vectors. CLIP embeddings align images and text in a shared space, enabling zero-shot classification via text prompts. DINO/DINOv2 produce self-supervised features useful for downstream tasks without labels. These embeddings power image retrieval, similarity search, and as input to other ML models.
Data Augmentation and MLOps for Vision
Data augmentation expands training data diversity. Geometric augmentations include random crop, flip, rotation, and scaling. Color augmentations adjust brightness, contrast, saturation, and hue. Advanced techniques include CutMix (paste patches between images), MixUp (blend images and labels), RandAugment (random augmentation policy), and AutoAugment (learned policies). For production, tools like FiftyOne for dataset curation, Weights & Biases for experiment tracking, and ONNX/TensorRT for optimized inference are essential. Model serving options include TorchServe, TensorFlow Serving, and cloud services (AWS SageMaker, Vertex AI).
Worked examples
Example 1
A convolutional layer has an input image of size 224×224×3. It uses 64 filters of size 7×7 with stride 2 and padding 3. What is the output feature map dimension?
Solution
Step 1: Identify the given parameters
- Input spatial size (H, W) = 224×224
- Number of filters (output channels) = 64
- Kernel size (K) = 7×7
- Stride (S) = 2
- Padding
Step 2: Apply the convolution output formula
Output spatial size = (Input - Kernel + 2×Padding) / Stride + 1
= (224 - 7 + 2×3) / 2 + 1
= 111.5 + 1 → 112 (integer floor, common in frameworks)
Step 3: Determine final output dimension
- Spatial: 112×112
- Channels: 64 (number of filters)
Answer: Output feature map dimension is 112×112×64
Example 2
An object detector predicts a bounding box [50, 50, 200, 200] for a car. The ground truth box is [60, 60, 190, 190]. Calculate the IoU. Should this prediction be considered correct at IoU threshold 0.5?
Solution
Step 1: Identify coordinates (x1, y1, x2, y2 format)
- Predicted box: (50, 50, 200, 200)
- Ground truth: (60, 60, 190, 190)
Step 2: Calculate intersection area
- Intersection x1 = max(50, 60) = 60
- Intersection y1 = max(50, 60) = 60
- Intersection x2 = min(200, 190) = 190
- Intersection y2 = min(200, 190) = 190
- Intersection width = 190 - 60 = 130
- Intersection height = 190 - 60 = 130
- Intersection area = 130 × ,900
Step 3: Calculate union area
- Predicted box area = (200-50) × × ,500
- Ground truth area = (190-60) × × ,900
- Union = Predicted + Ground truth - Intersection
Step 4: Calculate IoU
IoU = Intersection / Union = 16,900 / 22,.751
Answer: .751 (75.1%). Since 0.751 > 0.5, this prediction IS considered correct at the 0.5 IoU threshold.
Example 3
You have 5,000 labeled images for defect detection in manufacturing. You want to use transfer learning with a pretrained ResNet50. Design a training strategy.
Solution
Step 1: Split the data
- Training: 4,000 images (80%)
- Validation: 500 images (10%)
- Test: 500 images (10%)
- Apply stratified splitting to maintain class balance
Step 2: Data augmentation strategy
- Use albumentations or torchvision.transforms
- Augmentations: horizontal flip, rotation (±15°), brightness/contrast adjustment
- Consider CutMix or MixUp if classes are imbalanced
- For defect detection, preserve defect integrity-avoid aggressive crops that might remove defects
Step 3: Model modification
- Load ResNet50 with pretrained ImageNet weights (via torchvision or timm)
- Replace final fully connected layer: nn.Linear(2048, num_classes)
- Initialize new classifier weights appropriately
Step 4: Training phases
Phase 1 (Frozen backbone):
- Freeze all ResNet layers except the classifier head
- Train for 5-10 epochs with .01
- This quickly adapts the classifier to your classes
Phase 2 (Fine-tuning):
- Unfreeze the last 2-3 residual blocks
- Use discriminative learning rates: backbone , classifier lr=1e-3
- Optimize using AdamW with weight decay (1e-2), initial linear warmup over the first 3 epochs, and cosine annealing schedule
- Train for 20-30 epochs with early stopping on validation loss
Step 5: Monitoring and optimization
- Track validation accuracy, F1 score (better for imbalanced data)
- Save best checkpoint based on validation F1
Answer: Use two-phase transfer learning-freeze backbone for initial classifier adaptation, then fine-tune last layers with discriminative learning rates, AdamW, and linear warmup with cosine decay. This approach balances fast convergence with preservation of learned features, suitable for the limited dataset size.
Example 4
Compare using CLIP embeddings versus a fine-tuned CNN classifier for a product image search system with 100,000 catalog images. Recommend the better approach.
Solution
Step 1: Analyze CLIP embeddings approach
Pros:
- Zero-shot capability: no training data needed
- Unified image-text space: enables natural language queries ('red summer dress')
- Fast deployment: precompute embeddings, use approximate nearest neighbor (ANN) search
- Generalization: handles unseen product categories
Cons:
- Lower accuracy on domain-specific products compared to fine-tuned models
- Fixed embedding size: may not capture fine-grained details (texture, material)
- Computational: requires GPU for embedding generation (though inference is fast)
Step 2: Analyze fine-tuned CNN approach
Pros:
- Higher accuracy on your specific product categories
- Can optimize for domain-specific features
- Smaller model possible via distillation or efficient architectures
Cons:
- Requires labeled training data (category labels)
- Training time and hyperparameter tuning
- Text queries need separate processing (no unified embedding space)
- Retraining needed for new categories
Step 3: Hybrid recommendation
For product search at this scale:
1. Use CLIP as the primary embedding model:
- Precompute 512-dim CLIP embeddings for all 100,000 images
- Build ANN index using FAISS or ScaNN
- Enable both image-to-image and text-to-image search
2. For critical product categories with high search volume:
- Fine-tune a smaller model (MobileNet, EfficientNet-B0) for reranking
- Use CLIP for retrieval, then rerank top-100 results with fine-tuned model
3. Implementation:
- CLIP ViT-B/32: fast inference, good quality
- FAISS IVF with 256 clusters: balance speed and recall
- Approximate search: <10ms for 100K images
Answer: Recommend CLIP embeddings as the primary approach with optional fine-tuned reranking. This provides immediate deployment capability, natural language search, and good accuracy. The hybrid approach adds fine-tuned reranking for high-traffic categories, combining CLIP's flexibility with domain-specific accuracy. This balances development speed, search quality, and computational efficiency for the 100,000-image catalog.
Representative solved questions
See the kind of question in this topic before opening the full practice set.
Question 1
In the context of object detection, what is mean Average Precision (mAP)?
Answer: C. The mean of average precision values calculated across all classes at various IoU thresholds
ExplanationStep 1: mAP is the standard object detection metric.
Step 2: Precision is calculated at different recall values for each class.
Step 3: Average Precision (AP) is the area under the precision-recall curve.
Step 4: mAP averages AP across all classes, often at multiple IoU thresholds.
Answer: mAP is the mean of AP values across classes at various IoU thresholds.
Sources and review notes
This is an AISEA-authored practice question.
Review status: accepted · Reviewed 2026-08-13 · structure and answer-key checks, editorial quality checks, duplicate screening
Question 2
What is the difference between padding='same' and padding='valid' in convolution?
Answer: D. 'Same' adds zero padding to maintain input dimensions; 'valid' uses no padding, reducing output dimensions
ExplanationStep 1: Convolution reduces spatial dimensions without padding.
Step 2: Padding='valid' means no padding, output shrinks based on kernel size.
Step 3: Padding='same' adds zeros around the input.
Step 4: 'Same' padding keeps output dimensions equal to input dimensions.
Answer: 'Same' maintains dimensions with zero padding; 'valid' uses no padding.
Sources and review notes
This is an AISEA-authored practice question.
Review status: accepted · Reviewed 2026-08-13 · structure and answer-key checks, editorial quality checks, duplicate screening
Question 3
What is the bottleneck architecture in ResNet?
Answer: B. Using 1x1 convolutions to reduce and restore dimensions, sandwiching a 3x3 convolution for efficiency
ExplanationStep 1: Standard ResNet blocks use two 3x3 convolutions.
Step 2: Bottleneck blocks use 1x1, 3x3, 1x1 convolutions.
Step 3: The first 1x1 reduces channels (compression), 3x3 processes, 1x1 expands.
Step 4: This reduces parameters while maintaining representational capacity.
Answer: Bottleneck uses 1x1 to reduce/restore dimensions around a 3x3 convolution.
Sources and review notes
This is an AISEA-authored practice question.
Review status: accepted · Reviewed 2026-08-13 · structure and answer-key checks, editorial quality checks, duplicate screening
Common mistakes and useful habits
- When calculating CNN output dimensions, always verify with a quick tensor shape check in code-framework implementations may have subtle rounding differences that affect your calculations.
- For object detection evaluation, understand the difference between mAP@0.5 (Pascal VOC metric) and mAP@0.5:0.95 (COCO metric averaged across IoU thresholds)-modern benchmarks use the latter.
- In transfer learning, start with the smallest possible learning rate for fine-tuning. A good rule: backbone LR should be 10× smaller than classifier LR to avoid destroying pretrained features.
- When choosing between YOLO and Faster R-CNN, consider: YOLO for real-time applications (>30 FPS needed), Faster R-CNN for maximum accuracy when speed is secondary. Modern YOLO variants have largely closed the accuracy gap.
- For production vision systems using traditional detectors, implement confidence thresholding and non-maximum suppression (NMS) to prune overlapping boxes. When ultra-low inference latency is critical, evaluate NMS-free models such as YOLOv10 or RT-DETR which eliminate post-processing overhead.
- When using multimodal models (GPT-4o, Claude 3.5/3.7 Sonnet, Gemini 1.5/2.0 Flash) for document understanding, pre-process images to appropriate resolution (typically 768-1024px on longest side) to balance token usage, latency, and OCR fidelity.
Sources and review notes
Reviewed by AISEA Editorial Desk on 7 Nov 2026.
- Updated object detection architecture coverage to include NMS-free paradigms (YOLOv10, RT-DETR).
- Modernized frontier multimodal references to GPT-4o, Claude 3.5/3.7, and Gemini 1.5/2.0.
- Added AdamW warmup and cosine decay transfer learning best practices.
- YOLOv10: Real-Time End-to-End Object DetectionUltralytics Documentation · Checked 7 Nov 2026Supports: NMS-free dual label assignments and real-time latency benchmarks.
- Transfer Learning for Computer Vision TutorialPyTorch Documentation · Checked 7 Nov 2026Supports: Fine-tuning conventions, learning rate schedules, and backbone adaptation.
Ready to test your understanding?
Work through 120 questions with explanations after each answer.