07 - Computer Vision Tasks: Theory and Concepts

This notebook provides the theoretical foundation for understanding different Computer Vision tasks before diving into their practical implementation with Ultralytics YOLO.

Learning Objectives

By the end of this notebook, you will be able to:

• Distinguish between classification, detection, segmentation, pose estimation, and tracking

• Explain what bounding boxes represent and how IoU measures overlap

• Describe the purpose of Non-Maximum Suppression (NMS)

• Interpret common evaluation metrics: precision, recall, and mAP

• Choose the appropriate CV task for a given problem

• Understand basic annotation formats used in practice

---

1. The CV Task Taxonomy

Computer Vision tasks form a hierarchy of increasing complexity:

Task	Input	Output	Example Use Case
Classification	Image	Single label	“Is this a cat or dog?”
Object Detection	Image	Bounding boxes + labels	“Where are the cars in this image?”
Instance Segmentation	Image	Pixel masks per object	“Outline each person exactly”
Semantic Segmentation	Image	Pixel-wise class map	“Label every pixel as road, sky, building…”
Pose Estimation	Image	Keypoints (skeleton)	“Where are the person’s joints?”
Object Tracking	Video	Boxes + persistent IDs	“Follow person #3 across frames”

Visual Comparison

Classification:     Detection:          Segmentation:       Pose:
┌─────────────┐    ┌─────────────┐     ┌─────────────┐    ┌─────────────┐
│             │    │  ┌─────┐    │     │  ██████     │    │     ○       │
│   [cat]     │    │  │ cat │    │     │  ██████     │    │    /|\      │
│             │    │  └─────┘    │     │  ██████     │    │    / \      │
│  Label: cat │    │  ┌───┐      │     │    ███      │    │  keypoints  │
└─────────────┘    │  │dog│      │     └─────────────┘    └─────────────┘
                   └──┴───┴──────┘

1.1 Classification

What it does: Assigns a single label (or probability distribution over labels) to an entire image.

Output: {class_name: probability} or just the top class.

Limitations:

• Cannot tell you where objects are

• Cannot handle multiple objects of different classes

• Assumes one dominant subject per image

When to use: Quality control (defect/no-defect), medical screening (disease/healthy), content moderation.

1.2 Object Detection

What it does: Locates objects in an image and assigns each a class label.

Output: A list of detections, each containing:

• Bounding box: (x_min, y_min, x_max, y_max) or (x_center, y_center, width, height)

• Class label: Which category (e.g., “person”, “car”)

• Confidence score: How sure the model is (0.0 to 1.0)

Key concepts:

• A bounding box is the smallest rectangle that fully contains an object

• Multiple objects can be detected in one image

• Each detection is independent

When to use: Counting objects, finding specific items, surveillance, autonomous driving.

1.3 Instance Segmentation

What it does: Like detection, but instead of boxes, provides a pixel-level mask for each object.

Output: For each object:

• Mask: Binary image where 1 = object pixel, 0 = background

• Class label

• Confidence score

Key difference from semantic segmentation: Instance segmentation distinguishes between individual objects of the same class (person 1 vs person 2), while semantic segmentation only labels pixels by class.

When to use: Precise object boundaries needed, overlapping objects, medical imaging, photo editing.

1.4 Pose Estimation

What it does: Detects keypoints (joints, landmarks) on objects—typically humans.

Output: For each detected person/object:

• Keypoints: List of (x, y, confidence) for each landmark

• Skeleton connections: Which keypoints connect (e.g., shoulder→elbow→wrist)

Common keypoints for humans (COCO format, 17 points):

• Head: nose, eyes, ears

• Upper body: shoulders, elbows, wrists

• Lower body: hips, knees, ankles

When to use: Sports analytics, fitness apps, sign language recognition, fall detection, animation.

1.5 Object Tracking

What it does: Extends detection to video by maintaining consistent IDs across frames.

Output: For each frame:

• Same as detection (boxes, classes, scores)

• Track ID: Unique identifier that persists across frames

Challenge: The same person in frame 1 should have the same ID in frame 100, even if they were occluded in between.

Common algorithms: SORT, DeepSORT, ByteTrack, BoT-SORT

When to use: Traffic monitoring, retail analytics, sports tracking, security systems.

---

2. Core Detection Concepts

Since detection is foundational to segmentation, pose, and tracking, let’s understand its key concepts in depth.

2.1 Bounding Boxes

A bounding box defines a rectangular region containing an object.

Two common formats:

1. Corner format (Pascal VOC, COCO): (x_min, y_min, x_max, y_max)

   • Top-left corner: (x_min, y_min)

   • Bottom-right corner: (x_max, y_max)

2. Center format (YOLO): (x_center, y_center, width, height)

   • Often normalized to image dimensions (0.0 to 1.0)

Conversion (assuming image size W×H):

# Corner → Center
x_center = (x_min + x_max) / 2
y_center = (y_min + y_max) / 2
width = x_max - x_min
height = y_max - y_min

# Center → Corner
x_min = x_center - width / 2
y_min = y_center - height / 2
x_max = x_center + width / 2
y_max = y_center + height / 2

2.2 Intersection over Union (IoU)

IoU measures how much two boxes overlap. It’s the standard metric for comparing predicted boxes to ground truth.

        ┌───────────┐
        │   Box A   │
        │     ┌─────┼─────┐
        │     │ ∩   │     │
        └─────┼─────┘     │
              │   Box B   │
              └───────────┘

IoU = Area of Intersection (∩) / Area of Union (A ∪ B)

IoU values:

• IoU = 1.0: Perfect overlap (identical boxes)

• IoU = 0.0: No overlap at all

• IoU ≥ 0.5: Typically considered a “match” in evaluation

• IoU ≥ 0.75: Strict matching threshold

Why it matters:

• Used to match predictions to ground truth during evaluation

• Used in NMS to filter duplicate detections

• Higher IoU threshold = stricter evaluation

2.3 Non-Maximum Suppression (NMS)

Problem: Detection models often produce multiple overlapping boxes for the same object.

Solution: NMS filters out redundant detections, keeping only the best one.

Algorithm:

1. Sort all boxes by confidence score (highest first)

2. Take the highest-confidence box, add it to final results

3. Remove all remaining boxes that overlap with it (IoU > threshold)

4. Repeat steps 2-3 until no boxes remain

NMS threshold (typically 0.45-0.7):

• Lower threshold = more aggressive filtering (fewer boxes)

• Higher threshold = less filtering (more boxes, may have duplicates)

Before NMS:              After NMS:
┌─────────────┐          ┌─────────────┐
│ ┌─────┐     │          │             │
│ │┌────┼┐    │    →     │  ┌─────┐    │
│ └┼────┘│    │          │  │ 0.9 │    │
│  └─────┘    │          │  └─────┘    │
│ 0.9 0.7 0.6 │          │             │
└─────────────┘          └─────────────┘

---

3. Evaluation Metrics

How do we measure if a detection model is good?

3.1 Precision and Recall

Precision: Of all detections the model made, how many were correct?

Precision = True Positives / (True Positives + False Positives)
          = Correct Detections / All Detections

Recall: Of all objects that exist, how many did the model find?

Recall = True Positives / (True Positives + False Negatives)
       = Correct Detections / All Ground Truth Objects

Trade-off:

• High confidence threshold → High precision, low recall (few but accurate detections)

• Low confidence threshold → Low precision, high recall (many detections, some wrong)

Example:

• Ground truth: 10 cars in image

• Model detects: 8 boxes

• Correct detections (IoU ≥ 0.5): 6

• Precision = 6/8 = 0.75 (75% of detections were correct)

• Recall = 6/10 = 0.60 (found 60% of all cars)

3.2 Average Precision (AP) and mAP

AP (Average Precision): The area under the precision-recall curve for one class.

mAP (mean Average Precision): The average of AP across all classes.

mAP = (AP_class1 + AP_class2 + ... + AP_classN) / N

Common variants:

• mAP@0.5: Using IoU threshold of 0.5 (PASCAL VOC style)

• mAP@0.5:0.95: Average over IoU thresholds from 0.5 to 0.95 in steps of 0.05 (COCO style, stricter)

Interpreting mAP:

• mAP > 0.5: Decent model

• mAP > 0.7: Good model

• mAP > 0.85: Excellent model (on that specific dataset)

Note: mAP values are dataset-specific. A model with mAP=0.6 on COCO might have mAP=0.9 on a simpler custom dataset.

---

4. Practical Guidance

4.1 Choosing the Right Task

If you need to…	Use
Know if an image contains X	Classification
Count objects or find their locations	Detection
Get precise object boundaries	Instance Segmentation
Label every pixel in the image	Semantic Segmentation
Analyze human body positions	Pose Estimation
Follow objects across video frames	Tracking

Cost considerations:

• Classification: Cheapest to annotate (just labels)

• Detection: Moderate (draw boxes)

• Segmentation: Expensive (pixel-level masks)

• Pose: Very expensive (precise keypoints)

4.2 Speed vs. Accuracy Trade-offs

Models typically come in size variants:

Model Size	Parameters	Speed	Accuracy	Use Case
Nano (n)	~3M	Very fast	Lower	Edge devices, real-time
Small (s)	~10M	Fast	Moderate	Mobile, embedded
Medium (m)	~25M	Moderate	Good	General purpose
Large (l)	~50M	Slower	High	When accuracy matters
XLarge (x)	~100M	Slow	Highest	Offline processing

Decision factors:

• Real-time requirement? → Smaller model

• Edge device with limited memory? → Nano/Small

• Accuracy critical (medical, safety)? → Large/XLarge

• Batch processing overnight? → Use the biggest you can

4.3 Common Annotation Formats

YOLO format (one .txt file per image):

class_id x_center y_center width height
0 0.5 0.5 0.2 0.3
1 0.7 0.8 0.1 0.15

• Values normalized to 0-1

• One line per object

COCO format (single JSON file):

{
  "images": [{"id": 1, "file_name": "img.jpg", "width": 640, "height": 480}],
  "annotations": [{"id": 1, "image_id": 1, "category_id": 0, "bbox": [100, 100, 50, 80]}],
  "categories": [{"id": 0, "name": "person"}]
}

• bbox in corner format: [x_min, y_min, width, height]

• Pixel coordinates (not normalized)

Pascal VOC format (XML per image):

<annotation>
  <object>
    <name>person</name>
    <bndbox>
      <xmin>100</xmin><ymin>100</ymin>
      <xmax>150</xmax><ymax>180</ymax>
    </bndbox>
  </object>
</annotation>

---

5. Connection to Upcoming Notebooks

Now that you understand the theory, the following notebooks will show you how to apply these concepts:

Notebook	What You’ll Do
08 - Intro to Ultralytics	Load YOLO models, run basic inference
09 - Object Detection	Detect objects, tune confidence/IoU thresholds
10 - Segmentation & Pose	Get pixel masks and body keypoints
11 - Object Tracking	Track objects across video frames
12 - People Counting	Build a complete counting application

Each practical notebook will reference concepts from this theory section. When you see terms like “IoU threshold” or “NMS”, you’ll know exactly what they mean.

---

Recap

Key Takeaways

1. Task hierarchy: Classification → Detection → Segmentation/Pose → Tracking (increasing complexity)

2. Bounding boxes: Rectangles defined by corners (x_min, y_min, x_max, y_max) or center (x_c, y_c, w, h)

3. IoU: Measures box overlap (0 = none, 1 = perfect); used for evaluation and NMS

4. NMS: Filters duplicate detections by keeping highest-confidence boxes

5. Metrics: Precision (correctness), Recall (completeness), mAP (overall quality)

6. Trade-offs: Smaller models = faster but less accurate; choose based on your constraints

Checklist

• I can explain the difference between detection and segmentation

• I understand what IoU measures and typical threshold values

• I know why NMS is necessary and how it works

• I can interpret precision, recall, and mAP scores

• I know which task to choose for different problems

---

Ready for hands-on practice? Continue to the next notebook: Introduction to Ultralytics YOLO.