How to Train YOLOv8 Segmentation on a Custom Dataset

Leave a Comment / By /

Training YOLOv8 segmentation on a custom dataset allows you to detect objects and generate pixel-level masks that describe their exact shapes. Unlike standard object detection, which predicts rectangular bounding boxes, YOLOv8 instance segmentation predicts a separate mask for each detected object. The YOLOv8 family includes dedicated segmentation checkpoints such as yolov8n-seg.pt, yolov8s-seg.pt, yolov8m-seg.pt, yolov8l-seg.pt, and yolov8x-seg.pt, all of which support training, validation, inference, and export.

Table of Contents

Introduction to YOLOv8 Segmentation Training

YOLOv8 segmentation training is useful when a project needs more precise object boundaries than normal bounding-box detection can provide.

For example, a detection model may draw one rectangular box around a vehicle, while an instance segmentation model can identify the exact visible outline of that vehicle at the pixel level.

Custom segmentation training involves preparing labeled images, creating polygon-based mask annotations, configuring a dataset YAML file, selecting a YOLOv8 segmentation model, and training it using the Ultralytics CLI or Python API.

Ultralytics supports custom datasets and configurable training parameters such as epochs, image size, batch size, device selection, and other hyperparameters.

What Is YOLOv8 Segmentation?

YOLOv8 segmentation refers to the instance segmentation capability available in the YOLOv8 model family.

Instance segmentation performs two related tasks:

  1. identifying individual objects,
  2. predicting a segmentation mask for each detected object.

This means that if an image contains three cars, the model can identify three separate car instances and generate an individual mask for each one.

YOLOv8 officially provides dedicated -seg model variants for this task.

Object Detection vs Instance Segmentation

Object detection predicts the approximate location of an object using a rectangular bounding box.

A typical detection may contain:

class
confidence
x1
y1
x2
y2

Instance segmentation provides this detection information while also predicting the shape of the object.

For example:

Detection:
Car → Bounding Box

Segmentation:
Car → Bounding Box + Object Mask

A segmentation mask can follow irregular object boundaries instead of including background pixels inside a rectangle.

This makes segmentation useful when an application needs to understand exactly which pixels belong to each object.

How YOLOv8 Segmentation Predicts Object Masks

YOLOv8 segmentation processes an input image through its feature-extraction architecture and produces object predictions together with segmentation information.

Each detected instance receives a mask that identifies the region belonging to that object.

During training, the model learns these shapes from polygon annotations supplied in the dataset.

During inference, Ultralytics exposes segmentation results through its prediction results system, allowing masks and associated detections to be processed programmatically. Ultralytics Predict mode supports instance segmentation and multiple image, video, and streaming sources.

Requirements for Training YOLOv8 Segmentation

Before training a custom model, you need a working Ultralytics installation, properly annotated data, and enough hardware resources for the selected model and training configuration.

Install Python and Ultralytics

Install the Ultralytics package with pip:

pip install ultralytics

Then verify that it can be imported:

from ultralytics import YOLO

You can also check your installation using:

yolo checks

Ultralytics supports both command-line and Python-based training workflows.

Choose the Right YOLOv8 Segmentation Model

YOLOv8 provides five standard segmentation model sizes:

yolov8n-seg.pt
yolov8s-seg.pt
yolov8m-seg.pt
yolov8l-seg.pt
yolov8x-seg.pt

These represent Nano, Small, Medium, Large, and Extra Large variants. Ultralytics officially lists all five YOLOv8 segmentation checkpoints as supporting training, validation, inference, and export.

For initial testing, yolov8n-seg.pt is a practical starting point because it has lower computational requirements.

A larger model may provide more capacity but generally requires more GPU memory and processing time.

GPU and Hardware Requirements

A dedicated GPU is strongly recommended for practical segmentation training, especially with large datasets or high image resolutions.

Ultralytics automatically uses an available GPU when possible and can fall back to CPU training when a compatible GPU is not available.

Memory requirements depend on:

  • model size,
  • batch size,
  • image resolution,
  • dataset characteristics,
  • augmentation settings.

If GPU memory is limited, you can reduce the batch size, image size, or model size.

Prepare a Custom Dataset for YOLOv8 Segmentation

Dataset preparation is one of the most important parts of custom segmentation training.

Every object should have a correctly assigned class and a polygon that closely follows its visible boundaries.

Collect and Organize Training Images

Collect images that accurately represent the environment in which the model will eventually operate.

Include variation in:

  • object size,
  • viewing angle,
  • lighting,
  • background,
  • object orientation,
  • partial occlusion,
  • camera distance,
  • image quality.

A dataset containing many nearly identical images may produce a model that performs well on the training environment but poorly on new scenes.

Annotate Object Masks and Polygons

Each object instance needs a polygon annotation.

Instead of drawing only a rectangle, you define a sequence of points around the object’s boundary.

Conceptually:

Point 1
   ↓
Point 2
   ↓
Point 3
   ↓
...
   ↓
Point N

These polygon vertices are later stored in the YOLO segmentation label.

Ultralytics requires each segmentation polygon to contain at least three (x, y) points.

More polygon points can represent complicated object boundaries more accurately, but excessive unnecessary points can also make annotations larger and more difficult to maintain.

Split Data into Training and Validation Sets

Your dataset should contain separate training and validation subsets.

The training data is used to optimize model weights.

The validation data is used to measure how well the model performs on images it does not directly train on.

A common structure is:

dataset/
├── images/
│   ├── train/
│   └── val/
├── labels/
│   ├── train/
│   └── val/
└── data.yaml

Ultralytics’ segmentation dataset examples use matching image and label splits for training and validation.

YOLOv8 Segmentation Dataset Format

YOLOv8 instance segmentation uses one text label file for each image.

Each row in the label file corresponds to one segmented object instance.

Ultralytics defines the standard segmentation row as:

<class-index> <x1> <y1> <x2> <y2> ... <xn> <yn>

The polygon coordinates are normalized relative to the image dimensions.

Images and Labels Folder Structure

A standard custom dataset can look like:

custom_segmentation/
├── images/
│   ├── train/
│   │   ├── image001.jpg
│   │   └── image002.jpg
│   └── val/
│       ├── image101.jpg
│       └── image102.jpg
│
├── labels/
│   ├── train/
│   │   ├── image001.txt
│   │   └── image002.txt
│   └── val/
│       ├── image101.txt
│       └── image102.txt
│
└── data.yaml

The image and label files should share the same base filename.

For example:

images/train/car01.jpg
labels/train/car01.txt

Ultralytics uses one text label file per corresponding image.

YOLO Segmentation Label Format

Unlike object detection labels, segmentation labels do not store a bounding box followed by the polygon.

The segmentation row consists of:

class_id x1 y1 x2 y2 x3 y3 ... xn yn

For example:

0 0.310 0.420 0.390 0.300 0.510 0.320 0.590 0.470 0.490 0.610 0.340 0.590

The first value:

0

represents the class.

The remaining values are polygon vertex coordinates.

Each object in the image gets its own row.

Class IDs and Polygon Coordinates

Class IDs usually begin at zero.

For example:

0 = person
1 = car
2 = bicycle

A polygon is then represented using coordinate pairs:

x1 y1
x2 y2
x3 y3
...
xn yn

Coordinates are normalized to values between 0 and 1 relative to image width and height.

Unlike fixed-length bounding box labels, different segmentation instances can contain different numbers of polygon points. Ultralytics explicitly allows segmentation rows to vary in length.

Create the Dataset YAML File

The YAML file tells YOLOv8 where your images are located and which object classes exist.

A simple configuration may look like:

path: /datasets/custom_seg

train: images/train
val: images/val

names:
  0: person
  1: car

The dataset YAML is then passed to YOLO during training.

Add Train and Validation Paths

You can define a dataset root:

path: /datasets/custom_seg

and specify paths relative to it:

train: images/train
val: images/val

This makes the dataset configuration portable and easier to manage.

Ultralytics uses YAML files to define segmentation dataset paths and classes.

Define Segmentation Class Names

Use the names field to map numerical class IDs to readable names.

For a single-class dataset:

names:
  0: crack

For multiple classes:

names:
  0: car
  1: truck
  2: motorcycle

Every class ID used inside the text labels must correspond correctly to this class mapping.

How to Train YOLOv8 Segmentation

Once the dataset is correctly structured and the YAML file is ready, training can begin.

YOLOv8 segmentation supports both CLI and Python workflows.

Train YOLOv8 Segmentation Using the Command Line

A basic training command is:

yolo segment train model=yolov8n-seg.pt data=data.yaml epochs=100 imgsz=640

This command tells Ultralytics to:

  • use the segmentation task,
  • train a YOLOv8 Nano segmentation model,
  • load the custom dataset,
  • train for 100 epochs,
  • use an image size of 640.

The Ultralytics CLI follows the general structure:

yolo TASK MODE ARGS

where segment is the task and train is the mode.

Train YOLOv8 Segmentation Using Python

The equivalent Python code is:

from ultralytics import YOLO

model = YOLO("yolov8n-seg.pt")

model.train(
    data="data.yaml",
    epochs=100,
    imgsz=640
)

Loading a pretrained segmentation checkpoint is usually a useful starting point because the network already contains learned visual features.

YOLOv8 segmentation models officially support custom training through the Ultralytics training interface.

Set Epochs, Image Size, and Batch Size

Three important training parameters are:

epochs
imgsz
batch

For example:

yolo segment train model=yolov8s-seg.pt data=data.yaml epochs=150 imgsz=640 batch=16

epochs determines how many passes through the training dataset are performed.

imgsz controls the training image resolution.

batch determines how many images are processed together.

Larger batch sizes and image resolutions can require significantly more GPU memory, so these settings should be chosen according to available hardware.

Ultralytics exposes these and many other hyperparameters through its training interface.

Validate the Trained Segmentation Model

Validation helps determine whether the trained model works on images that were not used to update its weights.

Ultralytics provides a dedicated validation mode and reports metrics including mAP values.

Check Precision, Recall, and mAP

You can validate a trained model using:

yolo segment val model=runs/segment/train/weights/best.pt data=data.yaml

Or with Python:

from ultralytics import YOLO

model = YOLO("runs/segment/train/weights/best.pt")
metrics = model.val(data="data.yaml")

Important evaluation metrics can include:

  • precision,
  • recall,
  • mAP50,
  • mAP50-95.

Ultralytics Val mode provides multiple mAP-based metrics for evaluating model quality.

For segmentation, evaluate mask-related results rather than relying only on bounding-box performance.

Evaluate Mask Prediction Quality

Numerical metrics are useful, but visually inspect segmentation results as well.

Check whether masks:

  • follow object boundaries,
  • include too much background,
  • miss parts of objects,
  • merge nearby objects,
  • break one object into incorrect regions.

Poor masks can indicate annotation quality problems even when bounding boxes look acceptable.

Test YOLOv8 Segmentation After Training

Once validation results are acceptable, test the model on new images and videos that were not included in training or validation.

Ultralytics Predict mode supports images, videos, directories, streams, and other sources.

Run Segmentation on Images

Using the CLI:

yolo segment predict model=runs/segment/train/weights/best.pt source=test.jpg

Using Python:

from ultralytics import YOLO

model = YOLO("runs/segment/train/weights/best.pt")

results = model("test.jpg")

The result can contain detection boxes and instance segmentation masks.

Run Segmentation on Videos

For video:

yolo segment predict model=best.pt source=video.mp4

Or:

results = model("video.mp4", stream=True)

for result in results:
    masks = result.masks

Ultralytics supports streaming inference, which yields results sequentially and can reduce memory usage for long videos.

Use the Best Trained Weights for Inference

Training usually produces checkpoints such as:

best.pt
last.pt

For deployment and testing, best.pt is generally the checkpoint selected according to the monitored validation performance during training.

Load it using:

model = YOLO("best.pt")

You can then use it for prediction, validation, or export.

How to Improve YOLOv8 Segmentation Accuracy

Poor segmentation results are not always solved by selecting a larger model.

Dataset and annotation quality should usually be investigated first.

Improve Mask Annotation Quality

Segmentation models learn directly from polygon boundaries.

If annotations are inaccurate, the model may learn inaccurate shapes.

Look for:

  • polygons cutting through objects,
  • excessive background inside masks,
  • missing object regions,
  • inconsistent annotation rules,
  • poorly labeled overlapping objects.

Polygon quality is particularly important because segmentation aims to predict precise object boundaries.

Increase Dataset Diversity

Include images covering the conditions your model will face in production.

Useful variations include:

  • different camera angles,
  • close and distant objects,
  • different backgrounds,
  • shadows,
  • bright and dark lighting,
  • partially hidden objects,
  • crowded scenes.

A diverse training set helps improve generalization.

Tune Training Parameters

Important parameters to experiment with include:

epochs
imgsz
batch
lr0
optimizer
patience

For example, increasing imgsz may help when objects contain small boundary details.

However, larger input resolution requires more computation and GPU memory.

Ultralytics supports extensive training hyperparameter configuration through both CLI arguments and Python.

Use Data Augmentation Effectively

Data augmentation can expose the model to additional visual variation during training.

Depending on configuration, augmentation may modify properties such as:

  • scale,
  • translation,
  • color,
  • orientation,
  • image composition.

Augmentation should remain realistic for the target application.

Extremely aggressive transformations may create examples that do not resemble real-world input and can reduce training quality.

Common YOLOv8 Segmentation Training Problems

Most custom segmentation problems can be traced back to annotations, dataset paths, class mappings, or insufficient hardware resources.

Incorrect Polygon Annotations

Polygon labels must follow the required structure:

class x1 y1 x2 y2 ... xn yn

Each polygon must contain at least three coordinate pairs.

Common mistakes include:

  • forgetting the class ID,
  • using pixel coordinates instead of normalized coordinates,
  • using fewer than three points,
  • assigning the wrong class,
  • creating invalid polygon boundaries.

Visualize annotations before training whenever possible.

Missing or Invalid Label Files

Each annotated image should have a corresponding .txt label file with the same base filename.

For example:

images/train/example01.jpg
labels/train/example01.txt

Also check:

  • file extensions,
  • directory names,
  • train and validation paths,
  • empty or malformed annotation rows.

Ultralytics expects one corresponding text annotation file per labeled image in its segmentation format.

Low Segmentation Accuracy

Low accuracy can result from:

  • poor polygon quality,
  • insufficient data,
  • class imbalance,
  • repetitive images,
  • very small objects,
  • inadequate training,
  • unsuitable image resolution,
  • difficult backgrounds.

Review prediction visualizations before assuming the model architecture is the problem.

In many custom datasets, improving labels produces larger gains than simply increasing the number of epochs.

GPU Out-of-Memory Errors

If training produces a CUDA out-of-memory error, reduce GPU memory usage.

Start by lowering the batch size:

batch=8

If necessary, reduce the image resolution:

imgsz=512

or use a smaller model:

yolov8n-seg.pt

Ultralytics training supports configurable batch, image size, and device options, making it possible to adapt training to available hardware.

FAQs About Training YOLOv8 Segmentation

How do I train YOLOv8 segmentation on a custom dataset?

Prepare images and polygon segmentation labels, create a YAML file containing the training and validation paths and class names, load a YOLOv8 segmentation checkpoint, and run training.

For example:

yolo segment train model=yolov8n-seg.pt data=data.yaml epochs=100 imgsz=640

YOLOv8 segmentation checkpoints officially support training through the Ultralytics framework.

What annotation format does YOLOv8 segmentation use?

The standard Ultralytics segmentation format uses one text file per image and one row per object.

Each row follows:

<class-index> <x1> <y1> <x2> <y2> ... <xn> <yn>

The coordinates represent normalized polygon vertices, and every polygon must contain at least three coordinate pairs.

How many images are needed to train YOLOv8 segmentation?

There is no fixed number that works for every dataset.

A simple, visually consistent task may begin producing useful results with hundreds of well-annotated images, while complex projects may require thousands or substantially more.

The required amount depends on:

  • number of classes,
  • object diversity,
  • background diversity,
  • object size,
  • segmentation complexity,
  • expected real-world conditions.

Annotation quality and diversity are usually more important than simply reaching a particular image count.

Which YOLOv8 segmentation model should I use?

YOLOv8 provides:

yolov8n-seg.pt
yolov8s-seg.pt
yolov8m-seg.pt
yolov8l-seg.pt
yolov8x-seg.pt

For initial testing or limited hardware, yolov8n-seg.pt or yolov8s-seg.pt is a practical choice.

For applications where model capacity is more important than speed or resource use, larger variants can be evaluated. All five are officially listed as YOLOv8 instance segmentation models.

Can YOLOv8 segmentation detect multiple object classes?

Yes. A custom segmentation dataset can contain multiple classes.

For example:

names:
  0: person
  1: car
  2: bicycle
  3: motorcycle

Each polygon row begins with the class ID corresponding to the segmented object.

The model can then learn both object categories and individual masks.

How long does YOLOv8 segmentation training take?

Training time depends on:

  • dataset size,
  • selected model,
  • image size,
  • batch size,
  • number of epochs,
  • GPU performance,
  • storage and data-loading speed.

A Nano model on a small dataset may train much faster than an Extra Large model using high-resolution images and thousands of examples.

There is therefore no universal training duration for custom segmentation.

Can I train YOLOv8 segmentation without a GPU?

Yes. Ultralytics can train on CPU when no suitable GPU is available. The training documentation states that an available GPU is automatically selected when possible, otherwise training can fall back to CPU.

However, segmentation training is computationally intensive, so CPU training can be considerably slower.

For meaningful custom projects, especially with large datasets, GPU training is usually preferable.

Conclusion

Training YOLOv8 segmentation on a custom dataset requires accurately annotated polygon masks, a correct YOLO dataset structure, a dataset YAML file, and an appropriate YOLOv8 segmentation checkpoint.

YOLOv8 officially provides Nano, Small, Medium, Large, and Extra Large segmentation models, all supporting training, validation, inference, and export.

For dataset labels, each object is represented by a class ID followed by normalized polygon coordinates:

class x1 y1 x2 y2 ... xn yn

Each object gets its own annotation row, polygon lengths can vary, and each segmentation polygon must contain at least three (x, y) coordinate pairs.

Once the dataset is prepared, YOLOv8 segmentation can be trained through either the CLI or Python API. After training, evaluate both numerical metrics and the visual quality of predicted masks. For most custom segmentation projects, careful polygon annotation, diverse training images, and appropriate model and training settings are the most important factors for achieving reliable results.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top