How to Train YOLOv8 OBB on a Custom Dataset

Leave a Comment / By /

Training YOLOv8 OBB on a custom dataset allows the model to detect objects using rotated bounding boxes instead of standard horizontal rectangles. OBB stands for Oriented Bounding Box, and it is especially useful when objects can appear at different angles, such as ships, aircraft, vehicles, buildings, text regions, and objects in aerial or satellite images. YOLOv8 includes dedicated OBB models such as yolov8n-obb.pt, yolov8s-obb.pt, yolov8m-obb.pt, yolov8l-obb.pt, and yolov8x-obb.pt.

Table of Contents

Introduction to YOLOv8 OBB Training

Standard object detectors usually predict horizontal bounding boxes aligned with the image axes. This works well for many everyday objects, but it becomes inefficient when an object is strongly rotated.

For example, a long ship positioned diagonally may require a large horizontal rectangle containing significant background. An oriented bounding box can rotate with the ship and fit its actual orientation more closely.

YOLOv8 supports oriented detection as a dedicated task and can be trained using custom images annotated with four rotated box corners.

What Is YOLOv8 OBB?

YOLOv8 OBB is the oriented object detection variant of YOLOv8. Instead of limiting bounding boxes to horizontal and vertical edges, OBB detection allows rectangular boxes to rotate according to an object’s orientation.

A predicted OBB still contains normal detection information such as an object class and confidence score, but it also represents orientation.

Ultralytics internally works with an xywhr representation containing box center coordinates, width, height, and rotation, while YOLO OBB training labels use the four corners of the oriented rectangle.

Oriented Bounding Boxes vs Standard Bounding Boxes

A standard bounding box is axis-aligned:

┌───────────────────┐
│      Object       │
│       /////       │
│     //////        │
└───────────────────┘

The box itself cannot rotate.

An oriented bounding box can follow the object:

       /────────/
      / Object /
     /────────/

This often reduces the amount of irrelevant background included inside the box.

Standard detection usually stores a box as:

x_center y_center width height

An OBB additionally needs orientation information internally or can be represented through its four corner points.

When OBB Detection Is Useful

OBB detection is particularly useful when object orientation carries meaningful spatial information.

Common applications include:

  • aerial imagery,
  • satellite imagery,
  • ship detection,
  • aircraft detection,
  • rotated vehicles,
  • building footprints,
  • document and text regions,
  • industrial parts.

Ultralytics specifically highlights aerial and satellite imagery as important OBB use cases because objects frequently appear at different angles.

Requirements for Training YOLOv8 OBB

To train YOLOv8 OBB, you need the Ultralytics package, a correctly annotated oriented bounding box dataset, and enough processing resources for your selected model and training configuration.

Install Python and Ultralytics

Install Ultralytics using pip:

pip install ultralytics

You can verify the package from Python:

from ultralytics import YOLO

print("Ultralytics loaded successfully")

Ultralytics supports custom training through both Python and CLI interfaces.

Choose a YOLOv8 OBB Model

YOLOv8 provides five standard OBB model sizes:

yolov8n-obb.pt
yolov8s-obb.pt
yolov8m-obb.pt
yolov8l-obb.pt
yolov8x-obb.pt

These represent Nano, Small, Medium, Large, and Extra Large variants. All are listed by Ultralytics as supporting training, validation, inference, and export.

For a first custom experiment, yolov8n-obb.pt or yolov8s-obb.pt is usually a practical starting point because smaller models require fewer computational resources.

Larger models provide greater capacity but generally need more memory and inference time.

GPU and Hardware Requirements

YOLOv8 OBB can technically be trained without a dedicated GPU, but GPU acceleration is strongly recommended for practical training.

Hardware requirements depend on:

  • OBB model size,
  • image resolution,
  • batch size,
  • dataset size,
  • augmentation settings.

Aerial OBB datasets often use relatively high-resolution imagery, which can make memory requirements significant.

If GPU memory is limited, reduce the batch size, lower the image resolution, or choose a smaller YOLOv8 OBB model.

Prepare a Custom Dataset for YOLOv8 OBB

Dataset preparation is critical because the model needs to learn both object location and orientation.

Every target object should be labeled with a rotated rectangle that accurately follows its direction.

Collect and Organize Training Images

Collect images that represent the real conditions in which the model will operate.

Your dataset should include diversity in:

  • object angle,
  • scale,
  • location,
  • image quality,
  • background,
  • lighting,
  • object density,
  • partial occlusion.

For example, an aerial vehicle dataset should not contain vehicles facing only one direction. The model needs examples across many different rotations.

Annotate Rotated Objects

Each object should be labeled using an oriented rectangle rather than a normal horizontal bounding box.

The four corners should closely surround the object.

For a ship positioned diagonally, the OBB should rotate with the ship rather than leaving large background regions around it.

Annotation quality directly affects how accurately the model learns orientation.

Ultralytics Platform supports a dedicated OBB annotation tool for drawing oriented boxes.

Split Data into Training and Validation Sets

Separate your dataset into training and validation sets.

A common structure is:

custom_obb/
├── images/
│   ├── train/
│   └── val/
├── labels/
│   ├── train/
│   └── val/
└── data.yaml

The training split is used to optimize model weights, while the validation split measures generalization performance.

YOLOv8 OBB Dataset and Label Format

The Ultralytics YOLO OBB format stores one text annotation file per image.

Each object is represented using its class ID followed by four normalized corner coordinates.

Images and Labels Folder Structure

A typical OBB dataset looks like:

dataset/
├── images/
│   ├── train/
│   │   ├── image001.jpg
│   │   └── image002.jpg
│   └── val/
│       ├── image101.jpg
│       └── image102.jpg
│
├── labels/
│   ├── train/
│   │   ├── image001.txt
│   │   └── image002.txt
│   └── val/
│       ├── image101.txt
│       └── image102.txt
│
└── data.yaml

Image and label files should use matching base filenames.

For example:

images/train/ship001.jpg
labels/train/ship001.txt

Oriented Bounding Box Annotation Format

The official Ultralytics OBB label format is:

class_index x1 y1 x2 y2 x3 y3 x4 y4

For example:

0 0.20 0.30 0.45 0.20 0.60 0.42 0.35 0.52

Here:

0

is the class ID.

The remaining eight values represent the four corners:

x1 y1
x2 y2
x3 y3
x4 y4

Each object gets one annotation line.

If an image contains three objects, its label file contains three lines.

Normalized Corner Coordinates

OBB corner coordinates are normalized between 0 and 1.

For an image with width W and height H:

normalized_x = pixel_x / W
normalized_y = pixel_y / H

For example, if the image size is:

1000 × 500

and one corner is located at:

x = 250
y = 100

the normalized coordinate becomes:

x = 0.25
y = 0.20

The label stores:

0.25 0.20

Internally, Ultralytics converts oriented boxes into an xywhr representation containing center x, center y, width, height, and rotation.

Create the Dataset YAML File

The dataset YAML file tells YOLO where the training images are stored and what object classes are present.

Define Training and Validation Paths

A simple YAML file might look like:

path: /datasets/custom_obb

train: images/train
val: images/val

You can also add a test path when needed:

test: images/test

The paths can be relative to the main dataset root.

Add Object Class Names

Define your classes using the names field.

For example:

names:
  0: plane
  1: ship
  2: vehicle

A complete example is:

path: /datasets/custom_obb

train: images/train
val: images/val

names:
  0: plane
  1: ship
  2: vehicle

Every class index used in the .txt labels must correspond to the correct entry in this mapping.

How to Train YOLOv8 OBB

Once your images, OBB labels, and YAML configuration are ready, training can begin.

Ultralytics supports custom training through both Python and CLI interfaces.

Train YOLOv8 OBB Using the Command Line

A basic CLI command is:

yolo obb train model=yolov8n-obb.pt data=data.yaml epochs=100 imgsz=640

This configuration:

  • selects the OBB task,
  • loads the YOLOv8 Nano OBB checkpoint,
  • uses your custom dataset,
  • trains for 100 epochs,
  • uses an image size of 640.

You can replace the Nano model with another YOLOv8 OBB variant when more capacity is required.

Train YOLOv8 OBB Using Python

You can train the same model from Python:

from ultralytics import YOLO

model = YOLO("yolov8n-obb.pt")

results = model.train(
    data="data.yaml",
    epochs=100,
    imgsz=640
)

Pretrained weights are generally a useful starting point because they provide existing visual features that can be adapted to the custom OBB dataset.

Configure Epochs, Image Size, and Batch Size

Important parameters include:

epochs
imgsz
batch
device
workers
patience
lr0
optimizer

For example:

yolo obb train model=yolov8s-obb.pt data=data.yaml epochs=150 imgsz=1024 batch=8

A higher image resolution can be particularly useful for aerial imagery containing small objects.

However, increasing imgsz increases GPU memory consumption and computational cost.

Training arguments are configurable through the Ultralytics training interface.

Validate and Test the Trained OBB Model

Training loss alone is not enough to determine model quality.

You should validate the model and visually inspect predictions on unseen images.

Evaluate OBB Training Metrics

After training, load the best checkpoint:

from ultralytics import YOLO

model = YOLO("runs/obb/train/weights/best.pt")
metrics = model.val(data="data.yaml")

Ultralytics exposes OBB validation metrics including:

metrics.box.map
metrics.box.map50
metrics.box.map75
metrics.box.maps

These represent mAP50-95, mAP50, mAP75, and per-class mAP values.

Validation also helps identify whether the model performs consistently across different classes and orientations.

Run Predictions on New Images

Using CLI:

yolo obb predict model=runs/obb/train/weights/best.pt source=test.jpg

Using Python:

from ultralytics import YOLO

model = YOLO("runs/obb/train/weights/best.pt")

results = model("test.jpg")

OBB predictions can be accessed using:

for result in results:
    boxes = result.obb.xyxyxyxy
    classes = result.obb.cls
    confidence = result.obb.conf

Ultralytics exposes both four-corner OBB coordinates and xywhr representations in prediction results.

Test Rotated Object Detection on Videos

You can also test the model on videos:

yolo obb predict model=best.pt source=video.mp4

Or in Python:

results = model("video.mp4", stream=True)

for result in results:
    if result.obb is not None:
        print(result.obb.xyxyxyxy)

This is useful for monitoring moving vehicles, ships, aircraft, or industrial objects whose orientation changes over time.

How to Improve YOLOv8 OBB Accuracy

OBB performance depends heavily on annotation consistency and dataset diversity.

Improve Rotated Box Annotations

Make sure each oriented box closely fits its object.

Avoid:

  • excessive background,
  • boxes that cut through the object,
  • inconsistent corner placement,
  • horizontal boxes used for strongly rotated objects,
  • missing objects.

OBB annotations should consistently represent the object’s orientation.

Annotation errors can directly teach the model incorrect rotation patterns.

Increase Dataset Diversity

Include examples across many rotations.

For example, if training aircraft detection, include aircraft pointing:

north
south
east
west
diagonally
and intermediate angles

Also vary:

  • image scale,
  • background,
  • lighting,
  • object density,
  • camera altitude.

A model cannot reliably generalize to orientations or environments it never sees during training.

Tune Training Parameters

Experiment with:

epochs
imgsz
batch
augmentation
learning rate
model size

Small aerial objects may benefit from a larger image size because additional detail is preserved.

However, increasing resolution or model size should be tested rather than assumed to always improve performance.

Handle Small and Overlapping Objects

Small and densely packed rotated objects are particularly difficult.

Examples include:

  • cars in parking lots,
  • ships in crowded ports,
  • aircraft at airports,
  • containers.

Possible improvements include:

  • higher-resolution training,
  • more small-object examples,
  • accurate box placement,
  • larger datasets,
  • stronger but realistic augmentation.

For debugging an OBB pipeline before using a full dataset, Ultralytics provides small OBB datasets such as DOTA8 and DOTA128.

Common YOLOv8 OBB Training Problems

Most OBB training issues come from incorrect labels, dataset configuration, limited data, or hardware constraints.

Incorrect OBB Labels

Each annotation row must follow:

class_index x1 y1 x2 y2 x3 y3 x4 y4

Common problems include:

  • missing corner coordinates,
  • using standard xywh labels,
  • using raw pixel values instead of normalized coordinates,
  • incorrect class IDs,
  • boxes not matching object orientation.

A standard detection label like:

0 0.5 0.5 0.3 0.2

is not a valid OBB label because it lacks the four corner points.

Wrong Dataset YAML Configuration

Check:

  • dataset root path,
  • training path,
  • validation path,
  • class names,
  • class IDs.

For example, if labels contain class ID 2 but the YAML only defines classes 0 and 1, the dataset configuration is incorrect.

Low Detection Accuracy

Low accuracy may result from:

  • poor box annotations,
  • too little training data,
  • insufficient rotation diversity,
  • small object size,
  • incorrect class labels,
  • inadequate training,
  • unsuitable image resolution.

Visualize both labels and predictions before immediately changing model architecture or hyperparameters.

GPU Out-of-Memory Errors

If training runs out of GPU memory, reduce:

batch
imgsz
model size

For example:

yolo obb train model=yolov8n-obb.pt data=data.yaml epochs=100 imgsz=640 batch=4

A smaller batch is usually the first setting to adjust.

FAQs About Training YOLOv8 OBB

What is OBB in YOLOv8?

OBB stands for Oriented Bounding Box. It allows YOLOv8 to predict rotated rectangles around objects instead of limiting boxes to horizontal alignment.

This is especially useful for aerial, satellite, document, and industrial imagery where objects commonly appear at different angles.

How do I train YOLOv8 OBB on a custom dataset?

Prepare custom images with oriented bounding box labels, create a dataset YAML file, and train a YOLOv8 OBB checkpoint.

For example:

yolo obb train model=yolov8n-obb.pt data=data.yaml epochs=100 imgsz=640

YOLOv8 OBB models officially support custom training.

What annotation format does YOLOv8 OBB use?

The official format is:

class_index x1 y1 x2 y2 x3 y3 x4 y4

The four corner coordinates are normalized between 0 and 1.

Can YOLOv8 OBB detect rotated objects?

Yes. Rotated object detection is the main purpose of YOLOv8 OBB.

It can predict oriented bounding boxes for objects appearing at different angles and return class labels and confidence scores for each detection.

Which YOLOv8 OBB model should I use?

Available YOLOv8 OBB checkpoints include:

yolov8n-obb.pt
yolov8s-obb.pt
yolov8m-obb.pt
yolov8l-obb.pt
yolov8x-obb.pt

Start with Nano or Small for experimentation and limited hardware. Test Medium, Large, or Extra Large if additional model capacity is required and sufficient GPU resources are available.

How many images are needed for YOLOv8 OBB training?

There is no fixed minimum.

A relatively simple detection task may begin producing useful results with hundreds of high-quality examples, while complex aerial datasets may require thousands or many more.

More important factors include:

  • annotation accuracy,
  • rotation diversity,
  • class diversity,
  • object scale,
  • background variation.

A smaller diverse dataset can often be more useful than a larger collection of nearly identical images.

Can YOLOv8 OBB be trained without a GPU?

Yes, custom Ultralytics models can be trained on CPU, but OBB training can be computationally expensive.

CPU training is usually practical only for small experiments or debugging.

For larger datasets, higher image resolutions, and larger YOLOv8 OBB variants, GPU acceleration is strongly preferred.

Conclusion

Training YOLOv8 OBB on a custom dataset allows you to detect objects whose orientation cannot be represented efficiently with normal axis-aligned boxes.

YOLOv8 provides dedicated OBB checkpoints ranging from yolov8n-obb.pt to yolov8x-obb.pt, and these models support training, validation, inference, and export.

The most important dataset requirement is the OBB label format:

class_index x1 y1 x2 y2 x3 y3 x4 y4

where each pair represents one normalized corner of the rotated rectangle.

For reliable results, focus on accurate rotated-box annotations, a diverse range of object angles, sufficient examples of small and overlapping objects, and training settings appropriate for your available hardware. OBB is particularly valuable for aerial imagery, satellite analysis, ships, aircraft, rotated vehicles, buildings, and other applications where object orientation matters.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top