Computer Vision
Image understanding, classification, detection
Computer Vision (CV) enables machines to interpret and understand visual information from the world. It spans classical techniques like edge detection and feature extraction to modern deep learning approaches using Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). Today's multimodal models like GPT-4V and Claude 3 Vision can analyze images alongside text, opening new possibilities for visual reasoning and understanding.
Key formulas and rules
Key concepts
Convolutional Neural Networks (CNNs)
CNNs are the foundational architecture for image processing. They use learnable filters (kernels) that slide across input images to extract hierarchical features-from edges and textures in early layers to complex objects in deeper layers. Key components include convolutional layers (feature extraction), pooling layers (spatial downsampling), batch normalization (training stability), and fully connected layers (classification). Popular architectures include ResNet (residual connections), EfficientNet (compound scaling), and MobileNet (depthwise separable convolutions for efficiency).
Object Detection Architectures
Object detection localizes and classifies multiple objects in an image. Two-stage detectors like R-CNN, Fast R-CNN, and Faster R-CNN first propose regions of interest, then classify them-achieving higher accuracy but slower inference. Single-stage detectors like YOLO (You Only Look Once) and SSD (Single Shot Detector) predict bounding boxes and classes in one pass, offering real-time performance. Modern variants include YOLOv8/v9/v10, YOLO-NAS, and RT-DETR (real-time DETR with transformers). Key metrics include mAP@0.5, mAP@0.5:0.95, and inference FPS.
Image Segmentation
Segmentation assigns a class to each pixel. Semantic segmentation treats all instances of a class identically (e.g., all 'car' pixels merged), using architectures like U-Net, DeepLabv3+, and SegFormer. Instance segmentation distinguishes individual objects, combining detection with pixel masks-Mask R-CNN is the classic approach, with newer models like YOLOv8-seg and Mask2Former. Panoptic segmentation unifies both, assigning each pixel to either a semantic class or a specific instance identifier.
Vision Transformers (ViT)
Vision Transformers apply the transformer architecture to images by splitting them into patches, linearly embedding each patch, adding position embeddings, and processing through standard transformer encoder blocks. ViTs excel with large-scale pretraining (JFT-300M, ImageNet-21K) and can outperform CNNs on large datasets. Variants include DeiT (data-efficient training), Swin Transformer (hierarchical with shifted windows), and ViT-22B (largest vision model). ViTs lack inductive bias for translation invariance, requiring more data but offering better scalability and global context modeling.
Transfer Learning and Fine-tuning
Transfer learning leverages pretrained models on large datasets (ImageNet, LAION) for downstream tasks with limited data. Common approaches include feature extraction (freeze backbone, train classifier head), fine-tuning (unfreeze and train with lower learning rate), and progressive unfreezing. Pretrained weights are available via torchvision, timm (PyTorch Image Models), Hugging Face, and TensorFlow Hub. For vision tasks, even small datasets (100-1000 images) can achieve strong results with proper transfer learning and data augmentation.
Worked examples
Example 1
A convolutional layer has an input image of size 224×224×3. It uses 64 filters of size 7×7 with stride 2 and padding 3. What is the output feature map dimension?
Step 1: Identify the given parameters
- Input spatial size (H, W) = 224×224
- Number of filters (output channels) = 64
- Kernel size (K) = 7×7
- Stride (S) = 2
- Padding (P) = 3
Step 2: Apply the convolution output formula
Output spatial size = (Input - Kernel + 2×Padding) / Stride + 1
= (224 - 7 + 2×3) / 2 + 1
= (224 - 7 + 6) / 2 + 1
= 223 / 2 + 1
= 111.5 + 1 → 112 (integer floor, common in frameworks)
Step 3: Determine final output dimension
- Spatial: 112×112
- Channels: 64 (number of filters)
Answer: Output feature map dimension is 112×112×64
Example 2
An object detector predicts a bounding box [50, 50, 200, 200] for a car. The ground truth box is [60, 60, 190, 190]. Calculate the IoU. Should this prediction be considered correct at IoU threshold 0.5?
Step 1: Identify coordinates (x1, y1, x2, y2 format)
- Predicted box: (50, 50, 200, 200)
- Ground truth: (60, 60, 190, 190)
Step 2: Calculate intersection area
- Intersection x1 = max(50, 60) = 60
- Intersection y1 = max(50, 60) = 60
- Intersection x2 = min(200, 190) = 190
- Intersection y2 = min(200, 190) = 190
- Intersection width = 190 - 60 = 130
- Intersection height = 190 - 60 = 130
- Intersection area = 130 × 130 = 16,900
Step 3: Calculate union area
- Predicted box area = (200-50) × (200-50) = 150 × 150 = 22,500
- Ground truth area = (190-60) × (190-60) = 130 × 130 = 16,900
- Union = Predicted + Ground truth - Intersection
= 22,500 + 16,900 - 16,900 = 22,500
Step 4: Calculate IoU
IoU = Intersection / Union = 16,900 / 22,500 = 0.751
Answer: IoU = 0.751 (75.1%). Since 0.751 > 0.5, this prediction IS considered correct at the 0.5 IoU threshold.
Example 3
You have 5,000 labeled images for defect detection in manufacturing. You want to use transfer learning with a pretrained ResNet50. Design a training strategy.
Step 1: Split the data
- Training: 4,000 images (80%)
- Validation: 500 images (10%)
- Test: 500 images (10%)
- Apply stratified splitting to maintain class balance
Step 2: Data augmentation strategy
- Use albumentations or torchvision.transforms
- Augmentations: horizontal flip, rotation (±15°), brightness/contrast adjustment
- Consider CutMix or MixUp if classes are imbalanced
- For defect detection, preserve defect integrity-avoid aggressive crops that might remove defects
Step 3: Model modification
- Load ResNet50 with pretrained ImageNet weights (via torchvision or timm)
- Replace final fully connected layer: nn.Linear(2048, num_classes)
- Initialize new classifier weights appropriately
Step 4: Training phases
Phase 1 (Frozen backbone):
- Freeze all ResNet layers except the classifier head
- Train for 5-10 epochs with lr=0.01
- This quickly adapts the classifier to your classes
Phase 2 (Fine-tuning):
- Unfreeze the last 2-3 residual blocks
- Use discriminative learning rates: backbone lr=1e-4, classifier lr=1e-3
- Train for 20-30 epochs with early stopping on validation loss
Step 5: Monitoring and optimization
- Track validation accuracy, F1 score (better for imbalanced data)
- Use learning rate scheduling (cosine annealing or ReduceLROnPlateau)
- Save best checkpoint based on validation F1
Answer: Use two-phase transfer learning-freeze backbone for initial classifier adaptation, then fine-tune last layers with discriminative learning rates. This approach balances fast convergence with preservation of learned features, suitable for the limited dataset size.
Representative solved questions
See the kind of question in this topic before opening the full practice set.
Question 1
What is the primary function of a convolutional layer in a CNN?
Apply non-linear activation functions to the input
Extract spatial features from input images through learned filters
Store the weights for fully connected layers
Reduce the dimensionality of the image by pooling operations
Answer: B. Extract spatial features from input images through learned filters
ExplanationStep 1: Understand CNN architecture basics.
Step 2: Convolutional layers apply learned filters (kernels) across the input image.
Step 3: These filters detect features like edges, textures, and patterns.
Step 4: Each filter produces a feature map highlighting where specific features occur.
Answer: Convolutional layers extract spatial features through learned filters.
Sources and review notes
This is an AISEA-authored practice question.
Review status: accepted · Reviewed 2026-08-13 · structure and answer-key checks, editorial quality checks, duplicate screening
Question 2
In object detection, what does the IoU (Intersection over Union) metric measure?
The inference speed of the detection model
The overlap between predicted bounding box and ground truth bounding box
The number of objects detected per image
The confidence score threshold for detections
Answer: B. The overlap between predicted bounding box and ground truth bounding box
ExplanationStep 1: IoU is a standard metric in object detection.
Step 2: It calculates the area of intersection divided by the area of union.
Step 3: IoU = Area of Overlap / Area of Union.
Step 4: Higher IoU indicates better prediction accuracy.
Answer: IoU measures overlap between predicted and ground truth bounding boxes.
Sources and review notes
This is an AISEA-authored practice question.
Review status: accepted · Reviewed 2026-08-13 · structure and answer-key checks, editorial quality checks, duplicate screening
Common mistakes and useful habits
- When calculating CNN output dimensions, always verify with a quick tensor shape check in code-framework implementations may have subtle rounding differences that affect your calculations.
- For object detection evaluation, understand the difference between mAP@0.5 (Pascal VOC metric) and mAP@0.5:0.95 (COCO metric averaged across IoU thresholds)-modern benchmarks use the latter.
- In transfer learning, start with the smallest possible learning rate for fine-tuning. A good rule: backbone LR should be 10× smaller than classifier LR to avoid destroying pretrained features.
- When choosing between YOLO and Faster R-CNN, consider: YOLO for real-time applications (>30 FPS needed), Faster R-CNN for maximum accuracy when speed is secondary. Modern YOLO variants have largely closed the accuracy gap.
- For production vision systems, always implement confidence thresholding and non-maximum suppression (NMS). Without NMS, object detectors may produce multiple overlapping predictions for the same object.
- When using multimodal models (GPT-4V, Claude 3 Vision) for document understanding, pre-process images to appropriate resolution (typically 768-1024px on longest side) to balance token usage and OCR accuracy.
Ready to test your understanding?
Work through 120 questions with explanations after each answer.