SEMANTIC
What is each pixel?
All pixels of one class share one meaning. It understands “car,” not car 1 versus car 2.
pixel → class
π5 EXPLAINS · 02 · 5 MIN · 4 INTERACTIONS
Instance segmentation,
without the confusion.
A mask can tell you which pixels belong together without telling you what they are. That single distinction separates class-agnostic from class-aware instance segmentation.
Use the same street scene four ways. By the end, you will know which pixels, identities, and labels each segmentation task actually promises.
Image: humans see objects immediately, but the computer only receives a grid of colour values.
— classes
— identities
pixels coverage
00:00 · THREE QUESTIONS
Do you need a class,
an identity—or both?
Semantic segmentation asks “what class is each pixel?” Every car pixel receives the same car label, so two touching cars may become one region. Instance segmentation asks “which countable object owns each pixel?” It separates car 1 from car 2. A common benchmark also attaches a class, but the separation and the naming are logically different jobs.
Panoptic segmentation combines both views into one complete scene. Countable things such as people and cars get a class and an instance ID; amorphous stuff such as sky and road gets a class but normally no countable identity.
INSTANCE
Which object owns it?
Each countable object gets its own mask. A class label may be attached—but need not be.
pixel → instance ID
PANOPTIC
What, which, everywhere?
Every pixel receives a coherent answer: stuff gets a class; things get class plus identity.
pixel → class + ID
01:10 · THE COMMON CONFUSION
Identity is not the same
as category.
A class-aware system returns a semantic category with each instance: “person,” “car,” or “dog.” A class-agnostic system only needs to separate objects: “mask 1,” “mask 2,” and “mask 3.” It can leave naming to a later classifier—or never name the objects at all.
Class-agnostic does not mean “magically finds every unseen object.” Training data, prompts, thresholds, and the model’s notion of objectness still matter. It describes the output contract: instance boundaries are predicted or evaluated without requiring the correct semantic category.
Known vocabulary: masks carry labels from a fixed training set. The unfamiliar delivery robot may be missed or forced into the wrong class.
- OUTPUT
- mask + class + score
- NOVEL OBJECT
- may be missed
- SEMANTICS
- fixed vocabulary
Separate + name
Useful when the downstream action depends on category: count cars, inspect defects by type, or track pedestrians.
{ mask, class, score }
Separate first
Useful for object discovery, annotation tools, cells or parts with unknown taxonomies, and classify-later pipelines.
{ mask, score }
02:30 · HOW MASKS ARE PRODUCED
Different machinery.
The same mask-shaped goal.
Class-aware and class-agnostic are properties of the task or output—not a unique architecture. Detection-based, bottom-up, query-based, and promptable systems can all produce instance masks. What changes is how they propose and separate the objects.
03 / 04 · FOLLOW THE INFORMATION
DETECT → CROP → MASK
Top-down: find objects, then trace them.
A detector proposes a box for each likely object. A mask head predicts foreground pixels inside every proposal. Mask R-CNN is the classic example.
image features→object boxes→masks + labels
Class relation: often class-aware, although a shared class-agnostic mask head is also possible.
- 01
Top-down finds candidate objects first, then predicts a mask inside each region.
- 02
Bottom-up predicts pixel cues first, then groups pixels that appear to belong together.
- 03
Query-based predicts a set of mask-and-label pairs and assigns each object to one query.
- 04
Promptable turns a point, box, or prior mask into one or more plausible object masks.
03:45 · KEEP THE AXES SEPARATE
Three phrases that sound
more similar than they are.
Class-agnostic means category is not required for each mask. Open-vocabulary means categories can be supplied flexibly—often as text—instead of coming only from a fixed training list. Open-vocabulary output is still class-aware when it names the object.
Promptable describes the interaction. A model such as Segment Anything receives a point, box, or mask prompt and returns a class-agnostic mask. It does not attach a semantic name by itself, and prompting one object is not the same as automatically discovering every object in the image.
“Class-agnostic means open-vocabulary.”
FALSENo labels and flexible labels are different contracts.
“Every instance model must name the class.”
FALSEAn instance ID is enough to separate mask A from mask B.
“Class-agnostic guarantees unseen objects.”
FALSEGeneralisation still depends on data, objectness, and prompting.
THE EVALUATION CHANGES TOO
Same overlap. Different notion of correct.
IoU = 0.82The mask overlaps the object well.
CLASS-AWARE
Wrong label → miss
CLASS-AGNOSTIC
Class ignored → match
04:30 · START FROM THE APPLICATION
Ask what information
must survive downstream.
Do not choose from model names first. Choose the output contract. If the application only needs separate regions, class-agnostic masks avoid committing to a taxonomy. If it needs category-specific decisions, include semantics. If it needs complete scene coverage, move toward semantic or panoptic segmentation.
04 / 04 · CHOOSE A REQUIREMENT
RECOMMENDED OUTPUT
Class-aware instance segmentation
You need a separate mask for every vehicle and a category such as car, van, or bicycle for the count.
IDENTITY required
CLASS required
ALL PIXELS not required
ORIGINAL PAPERS · GO DEEPER
2017Mask R-CNN ↗ 2018Panoptic Segmentation ↗ 2021Mask2Former ↗ 2023Segment Anything ↗