YOLOv8 segmentation masks describe the pixel-level shape of each detected object rather than representing it only with a rectangular bounding box. A YOLOv8 segmentation result can include bounding boxes, class IDs, confidence scores, binary or probability-based mask data, and polygon coordinates for each detected instance. In the Ultralytics Python API, segmentation information is exposed through the Results object, especially result.boxes and result.masks.
Introduction to YOLOv8 Segmentation Masks and Output
YOLOv8 instance segmentation combines object detection with pixel-level object separation. Instead of only predicting where an object is located, the model also estimates which pixels belong to that particular instance.
This is useful when the exact object boundary matters. For example, a bounding box around a car includes background pixels around the vehicle, while a segmentation mask can follow the visible outline of the car much more closely.
Ultralytics processes segmentation predictions and returns both detection information and masks in the final prediction results.
What Are Segmentation Masks in YOLOv8?
A segmentation mask is a spatial representation that identifies which pixels belong to an individual detected object.
For each detected instance, the model generates a corresponding mask. This allows two nearby objects of the same class to receive separate masks rather than being represented as one shared region.
Instance segmentation therefore combines object classification, localization, and shape estimation.
Difference Between Bounding Boxes and Segmentation Masks
Bounding boxes represent objects using rectangles.
For example:
Bounding Box
┌────────────────┐
│ │
│ Object │
│ │
└────────────────┘
The rectangle often contains pixels that do not actually belong to the object.
A segmentation mask instead identifies the object’s visible shape:
█████
█████████
███████████
█████████
███
This provides more detailed spatial information.
Bounding boxes are useful when object location is sufficient, while segmentation masks are better suited to applications that require exact object regions, contours, area measurements, or foreground extraction.
How YOLOv8 Represents Object Shapes
Internally, YOLO-style instance segmentation does not simply predict every final full-resolution mask independently from scratch.
The segmentation architecture uses shared prototype masks together with per-instance mask coefficients. These components are combined to generate a separate mask for each detected object. Ultralytics’ segmentation loss and mask-processing code explicitly use prototype tensors and mask coefficients to construct instance masks.
After processing, the resulting masks are exposed through the normal Ultralytics Results API so users do not usually need to manually reconstruct them.
How YOLOv8 Generates Segmentation Masks
YOLOv8 segmentation combines the normal detection pipeline with an additional mask-prediction branch.
The model predicts object localization and classification information while also generating the information required to reconstruct instance masks.
Object Detection and Mask Prediction
The detector first identifies candidate objects and predicts information such as:
- bounding box location,
- object class,
- confidence score.
The segmentation branch additionally predicts mask-related information for these detections.
This allows each final object to be associated with both its normal detection output and its instance mask.
Ultralytics’ segmentation predictor extends the normal detection predictor specifically to handle both boxes and masks.
Prototype Masks and Mask Coefficients
A central part of the YOLO segmentation design is the use of mask prototypes.
The network generates a set of shared prototype feature maps. Each detected object also receives a set of mask coefficients.
Conceptually:
Prototype Masks
+
Instance Mask Coefficients
↓
Individual Object Mask
Ultralytics’ loss implementation describes predicted mask coefficients with shape (N, 32) and prototype masks with shape (32, H, W) in the standard mask-processing path. It combines them mathematically to produce individual predicted masks.
The exact raw tensor dimensions can vary by model or exported format, so application code should avoid assuming one universal raw-output shape.
Creating the Final Instance Mask
The model combines an object’s mask coefficients with the shared prototype masks.
Conceptually, the operation produces:
mask = coefficients × prototype features
The resulting mask is then processed relative to the object’s bounding box and image dimensions.
Ultralytics provides mask-processing functions that combine prototypes, coefficients, and bounding boxes before producing the final masks used in Results objects.
This architecture allows the network to efficiently generate separate masks for multiple objects without running an entirely independent segmentation network for every instance.
YOLOv8 Segmentation Output Format
For normal Python inference, Ultralytics returns a Results object for each processed image.
Segmentation results can contain:
- bounding boxes,
- class IDs,
- confidence scores,
- mask tensors,
- polygon coordinates.
These elements can be accessed through properties of the Results object.
Bounding Box Output
Bounding boxes are available through:
result.boxes
Pixel-space corner coordinates can be accessed using:
result.boxes.xyxy
A typical box is conceptually represented as:
[x1, y1, x2, y2]
where:
x1= left coordinate,y1= top coordinate,x2= right coordinate,y2= bottom coordinate.
Each detected instance’s mask corresponds to its associated detection result.
Class IDs and Confidence Scores
The predicted class IDs can be accessed with:
result.boxes.cls
Detection confidence scores can be accessed with:
result.boxes.conf
For example:
for box in result.boxes:
class_id = int(box.cls[0])
confidence = float(box.conf[0])
These values describe what object the model detected and how confident it is about the detection.
Segmentation Mask Output
Segmentation results are available through:
result.masks
The underlying mask tensor can be accessed with:
result.masks.data
Ultralytics’ Results API defines a Masks structure for storing segmentation data along with the original image shape.
Conceptually, if four objects are detected, mask data may contain four masks:
Mask 0 → Object 0
Mask 1 → Object 1
Mask 2 → Object 2
Mask 3 → Object 3
The exact tensor dimensions depend on prediction configuration and processing behavior.
Understanding YOLOv8 Mask Coordinates
YOLOv8 segmentation results can be represented either as mask arrays or polygon coordinates.
Polygon coordinates are useful when you need object contours rather than full binary mask matrices.
Polygon Coordinates
Ultralytics provides polygon representations through the Masks result object.
For pixel-space polygon coordinates, use:
result.masks.xy
Each item represents a polygon corresponding to one detected object.
Conceptually:
polygon = result.masks.xy[0]
might contain:
[
[x1, y1],
[x2, y2],
[x3, y3],
...
]
These coordinates trace the detected object’s contour.
Normalized and Pixel Coordinates
Ultralytics also provides normalized polygon coordinates.
Pixel coordinates:
result.masks.xy
Normalized coordinates:
result.masks.xyn
Normalized coordinates are expressed relative to the original image dimensions.
For example, a point at:
x = 320
y = 240
in a:
640 × 480
image corresponds approximately to:
x = 0.5
y = 0.5
in normalized form.
The Results API provides both xy and xyn representations for segmentation polygons.
Mask Size and Image Resolution
The raw mask tensor size does not always have to equal the original input image resolution.
Ultralytics provides a retina_masks inference option. When enabled, returned masks.data can match the original image size; when disabled, masks use the model’s inference-size representation.
This distinction is important if you are combining masks with the original image manually.
Always check the mask tensor dimensions before assuming they match the source image.
How to Access YOLOv8 Segmentation Results
A basic segmentation inference workflow can look like:
from ultralytics import YOLO
model = YOLO("yolov8n-seg.pt")
results = model("image.jpg")
result = results[0]
The resulting object contains the detected boxes and masks.
Accessing Bounding Boxes and Classes
For example:
boxes = result.boxes.xyxy
classes = result.boxes.cls
confidences = result.boxes.conf
You can loop through them:
for i in range(len(result.boxes)):
box = result.boxes.xyxy[i]
class_id = int(result.boxes.cls[i])
confidence = float(result.boxes.conf[i])
print(box, class_id, confidence)
The Results API provides structured access to prediction data rather than requiring users to manually decode the entire model output tensor.
Extracting Mask Data in Python
The mask tensors can be accessed using:
masks = result.masks.data
For example:
if result.masks is not None:
for mask in result.masks.data:
print(mask.shape)
To convert mask data to NumPy:
mask_array = result.masks.data.cpu().numpy()
This is useful when performing custom image processing, mask measurements, or OpenCV operations.
Ultralytics also provides examples for isolating segmented objects using segmentation masks and OpenCV.
Accessing Polygon Points
Pixel-space polygon coordinates can be accessed using:
polygons = result.masks.xy
Normalized polygon coordinates can be accessed using:
normalized_polygons = result.masks.xyn
For example:
for polygon in result.masks.xy:
print(polygon)
This is useful for exporting contours, measuring geometry, or converting predictions into another annotation format.
YOLOv8 Segmentation Output for Multiple Objects
Instance segmentation treats each detected object separately.
If several objects are found in one image, YOLOv8 returns a separate detection and mask for each instance.
Separate Masks for Each Detected Object
Suppose an image contains:
Person
Car
Dog
The segmentation output may conceptually contain:
Detection 0 → Person → Mask 0
Detection 1 → Car → Mask 1
Detection 2 → Dog → Mask 2
Even if two objects belong to the same category, they can still receive different masks.
For example:
Car 1 → Mask 0
Car 2 → Mask 1
Car 3 → Mask 2
This is the key difference between instance segmentation and semantic segmentation, where pixels of the same category may instead be represented as one class-level region. Ultralytics describes instance segmentation as identifying individual objects and segmenting each from the rest of the image.
Matching Masks with Classes and Bounding Boxes
The prediction order allows mask information to be associated with the corresponding detection.
A practical loop might look like:
for i in range(len(result.boxes)):
box = result.boxes.xyxy[i]
cls = int(result.boxes.cls[i])
conf = float(result.boxes.conf[i])
mask = result.masks.data[i]
print("Class:", cls)
print("Confidence:", conf)
print("Box:", box)
print("Mask:", mask.shape)
Maintaining this index relationship is important when processing multiple objects.
If mask i is accidentally matched with box j, downstream segmentation analysis will become incorrect.
Visualizing YOLOv8 Segmentation Masks
Ultralytics provides built-in visualization functions, so users do not need to manually draw every prediction.
Overlaying Masks on Images
The easiest method is:
annotated = result.plot()
The plot() method overlays prediction information such as boxes and masks on the original image and returns the annotated result as a NumPy array.
You can display it with OpenCV:
import cv2
cv2.imshow("Segmentation", annotated)
cv2.waitKey(0)
This is useful for visually checking model predictions.
Saving Segmentation Results
Inference results can also be saved using Ultralytics’ prediction workflow.
For example:
results = model("image.jpg", save=True)
Or you can save a plotted result manually:
import cv2
annotated = result.plot()
cv2.imwrite("segmented.jpg", annotated)
The plotting API produces an annotated NumPy image containing the visualized predictions.
If you want only the segmented object rather than the full annotated image, the mask can be combined with the original image and saved separately. Ultralytics provides an official guide for isolating individual segmentation objects and saving them with solid or transparent backgrounds.
Working with Video Segmentation Output
YOLOv8 segmentation can also process video sources.
For example:
results = model("video.mp4", stream=True)
for result in results:
boxes = result.boxes
masks = result.masks
Using stream=True returns results sequentially, which can be more memory-efficient for longer videos.
Each video frame can contain a different number of detected objects and masks.
For applications that require persistent object identities across frames, segmentation can be combined with tracking rather than treating every frame completely independently.
Common Issues with YOLOv8 Segmentation Output
Incorrect mask interpretation can result from low-confidence detections, low image resolution, coordinate confusion, or assumptions about tensor shapes.
Missing or Incomplete Masks
Masks may be missing or incomplete when:
- the object is heavily occluded,
- the object is extremely small,
- the object confidence is too low,
- the image quality is poor,
- the model was not trained adequately for that object,
- the object is outside the model’s known classes.
If:
result.masks
is None, no usable segmentation masks were returned for that result.
Always check:
if result.masks is not None:
...
before accessing mask properties.
Low-Quality Mask Boundaries
Mask boundaries may appear rough when objects are:
- very small,
- blurred,
- partially hidden,
- visually similar to the background,
- poorly annotated during training.
Training mask quality also depends strongly on the accuracy of polygon annotations.
Increasing image resolution may help with very fine boundaries, but it also increases computational requirements.
Incorrect Mask Coordinates
One common mistake is confusing:
result.masks.xy
with:
result.masks.xyn
xy contains polygon coordinates in image pixel space.
xyn contains normalized coordinates relative to the original image size.
Another potential issue is assuming that masks.data always matches the original image resolution. Its dimensions depend on prediction configuration, including the retina_masks setting.
Overlapping Object Masks
Two predicted object masks can sometimes overlap.
This may occur when:
- objects physically overlap,
- detections are ambiguous,
- boundaries are difficult to distinguish,
- objects belong to visually similar classes.
Because instance segmentation predicts objects individually, each mask is stored separately even if some pixels overlap.
Applications that require mutually exclusive pixel assignments may need additional post-processing logic.
FAQs About YOLOv8 Segmentation Masks and Output
What does YOLOv8 segmentation output contain?
YOLOv8 segmentation output can include bounding boxes, class IDs, detection confidence scores, individual segmentation masks, and polygon representations of those masks.
In the Ultralytics Python API, these are primarily available through:
result.boxes
result.masks
How are segmentation masks represented in YOLOv8?
Internally, YOLO-style segmentation models use prototype masks together with mask coefficients to construct individual instance masks.
After post-processing, users can access the final mask data through:
result.masks.data
Does YOLOv8 return polygon coordinates?
Yes. Processed segmentation results expose polygon coordinates.
Use:
result.masks.xy
for pixel coordinates and:
result.masks.xyn
for normalized polygon coordinates.
Can YOLOv8 generate a separate mask for each object?
Yes. YOLOv8 performs instance segmentation, so each detected object can receive its own mask.
For example, three detected cars can produce three separate masks rather than one combined car region.
How do I extract masks from YOLOv8 results?
A basic example is:
from ultralytics import YOLO
model = YOLO("yolov8n-seg.pt")
results = model("image.jpg")
result = results[0]
if result.masks is not None:
masks = result.masks.data
polygons = result.masks.xy
print(masks)
print(polygons)
The Masks object is part of the Ultralytics Results API.
Are YOLOv8 segmentation mask coordinates normalized?
Both representations are available.
Use:
result.masks.xy
for pixel-space polygon coordinates.
Use:
result.masks.xyn
for normalized polygon coordinates.
The underlying masks.data tensor is a mask representation rather than a list of normalized polygon coordinates.
Can YOLOv8 segmentation output be saved as an image?
Yes. You can use:
results = model("image.jpg", save=True)
or create an annotated output with:
annotated = result.plot()
and save the resulting NumPy image manually. Ultralytics also provides a workflow for extracting and saving individual segmented objects.
Conclusion
YOLOv8 segmentation masks and output provide substantially more spatial information than normal object detection. Along with bounding boxes, class IDs, and confidence scores, a segmentation model predicts an individual pixel-level mask for every detected object.
Internally, YOLO-style instance segmentation combines shared prototype masks with per-instance mask coefficients to create the final object masks.
In the Ultralytics Python API, the most useful segmentation properties are:
result.boxes.xyxy
result.boxes.cls
result.boxes.conf
result.masks.data
result.masks.xy
result.masks.xyn
The data property provides mask tensors, xy gives pixel-space polygon points, and xyn provides normalized polygon coordinates.
Understanding the relationship between boxes, classes, confidence scores, masks, and polygon coordinates makes it much easier to use YOLOv8 segmentation output for image analysis, object extraction, video processing, measurement, and custom computer vision applications.
I’m Jane Austen, a skilled content writer with the ability to simplify any complex topic. I focus on delivering valuable tips and strategies throughout my articles.