YOLOv8 Hand Pose Estimation: Detection, Keypoints, and Custom Training

YOLOv8 hand pose estimation uses pose/keypoint detection to locate hands and predict landmark positions representing the wrist, fingers, and individual finger joints. YOLOv8 supports pose training and inference, so it can be fine-tuned on a hand-keypoint dataset for applications such as gesture recognition, sign-language systems, human-computer interaction, robotics, and AR/VR. A common hand landmark structure contains 21 keypoints per hand, including the wrist and four landmarks for each finger.

Introduction to YOLOv8 Hand Pose Estimation

Hand pose estimation is more detailed than simply detecting whether a hand is present in an image. The goal is to locate the hand and estimate important landmark positions that describe the arrangement of the palm, fingers, and joints.

YOLOv8 supports pose/keypoint detection as one of its official computer vision tasks. Its pose variants include models such as yolov8n-pose.pt, yolov8s-pose.pt, yolov8m-pose.pt, yolov8l-pose.pt, and yolov8x-pose.pt.

The standard pretrained YOLOv8 Pose checkpoints are designed around human-body pose data rather than a dedicated 21-landmark hand layout. For detailed hand pose estimation, a YOLOv8 Pose model can instead be trained or fine-tuned using a hand-keypoint dataset with the desired landmark structure.

Ultralytics provides an official Hand Keypoints dataset containing 26,768 images and 21 annotated landmarks per hand, making it a useful reference format for building a hand pose model.

What Is Hand Pose Estimation in YOLOv8?

Hand pose estimation uses a pose model to identify a hand instance and estimate predefined keypoints associated with it.

Instead of returning only a rectangular bounding box, the trained pose model can provide coordinates for landmarks such as the wrist, thumb joints, fingertips, and joints of the index, middle, ring, and little fingers.

The resulting landmark layout represents the structure and orientation of the hand.

Hand Detection vs Hand Keypoint Detection

Hand detection only answers where a hand is located.

Its output may include:

• hand bounding box,
• class label,
• detection confidence.

Hand keypoint detection provides additional information about the internal hand structure.

For example, a pose model can return:

• wrist location,
• thumb tip,
• index fingertip,
• middle fingertip,
• ring fingertip,
• little-finger tip,
• intermediate finger joints.

This extra information makes pose estimation more useful for analyzing hand gestures and movement.

How YOLOv8 Pose Models Identify Hand Landmarks

A YOLOv8 Pose model first processes the image through its feature-extraction network. The pose detection head then generates object localization and keypoint predictions.

For a custom hand model, each detected hand is associated with a fixed number of landmarks defined during training.

If the model is trained with:

kpt_shape: [21, 3]

each hand contains 21 keypoints with three values associated with each landmark in the training format.

The model learns the spatial patterns connecting these landmarks from the annotated training examples.

How YOLOv8 Hand Pose Estimation Works

YOLOv8 hand pose estimation combines object localization with landmark prediction.

The model learns not only that an image region contains a hand but also how keypoints are spatially arranged inside that hand.

Detecting the Hand Region

The first important output is the location of the hand.

The pose model predicts a bounding box that identifies the area occupied by each hand instance.

For example, an image containing two visible hands can produce two detections, with each detection associated with its own bounding box and keypoint set.

This instance-level separation is important because landmarks from different hands must remain associated with the correct hand.

Predicting Hand Keypoints

After learning from annotated examples, the model predicts the location of every defined hand landmark.

With a 21-keypoint configuration, the output represents points across the wrist and five fingers. The official Ultralytics Hand Keypoints dataset uses exactly 21 landmarks per hand.

Each landmark has x and y positional information, and pose outputs can also provide a confidence value representing prediction certainty.

Generating the Final Hand Pose

The predicted keypoints can be connected according to a predefined skeleton structure.

For example:

Wrist
  ├── Thumb
  ├── Index Finger
  ├── Middle Finger
  ├── Ring Finger
  └── Little Finger

Within each finger, neighboring joints are connected to form a visual representation of the hand.

The model predicts the landmarks, while visualization or downstream application logic can connect them to create the final skeleton.

YOLOv8 Hand Keypoints and Skeleton

The exact number of hand landmarks is determined by the dataset used for training.

A commonly used configuration contains 21 keypoints, which is also the structure used by the official Ultralytics Hand Keypoints dataset.

Common Hand Landmark Structure

The 21-landmark structure consists of:

• 1 wrist point,
• 4 thumb points,
• 4 index-finger points,
• 4 middle-finger points,
• 4 ring-finger points,
• 4 little-finger points.

This gives:

1 + 4 + 4 + 4 + 4 + 4 = 21 keypoints

The official Ultralytics dataset uses the following ordered layout:

Finger and Joint Keypoints

Most fingers use several landmarks representing successive joints from the palm toward the fingertip.

For example, the index finger uses:

Index MCP
   ↓
Index PIP
   ↓
Index DIP
   ↓
Index Tip

Similar structures are used for the middle, ring, and little fingers.

The thumb uses its own joint structure:

Thumb CMC
   ↓
Thumb MCP
   ↓
Thumb IP
   ↓
Thumb Tip

Together, these landmarks allow the pose model to represent finger bending, spreading, pointing, and other hand configurations.

Keypoint Coordinates and Confidence Scores

Predicted keypoints contain image coordinates.

In the Ultralytics Results API, pose predictions can provide x and y coordinates along with confidence scores for landmarks.

Conceptually, one landmark may look like:

[x, y, confidence]

For example:

[312.4, 185.7, 0.94]

This represents the predicted position and confidence for a particular hand landmark.

Low-confidence points can be ignored or treated cautiously in applications where reliable joint positions are required.

Dataset Requirements for YOLOv8 Hand Pose Estimation

A custom hand pose model requires images containing hands together with accurate bounding box and landmark annotations.

Dataset quality has a major effect on the final model.

Collecting Hand Pose Images

A useful hand dataset should contain variation in:

• left and right hands,
• open and closed hands,
• different finger positions,
• different skin appearances,
• camera angles,
• hand rotations,
• object interactions,
• backgrounds,
• lighting,
• image resolutions,
• partial occlusions.

Training only on clear, front-facing open hands can result in poor generalization when the model encounters real-world hand positions.

The official Ultralytics Hand Keypoints dataset contains 26,768 images, divided into 18,776 training images and 7,992 validation images.

Annotating Hand Keypoints

Every hand should be labeled consistently.

If you use the 21-point layout, landmark index 0 must always represent the wrist, index 4 must always represent the thumb tip, index 8 must always represent the index fingertip, and so on.

The same ordering must be maintained across the complete dataset.

Poorly placed landmarks or inconsistent ordering can make it difficult for the model to learn correct finger geometry.

Preparing YOLO Pose Labels

YOLO Pose labels extend normal detection annotations with keypoint information.

A pose annotation conceptually follows:

class x_center y_center width height kp1_x kp1_y kp1_v ... kpN_x kpN_y kpN_v

For a 21-point hand model, the dataset configuration may use:

kpt_shape: [21, 3]

The official Ultralytics Hand Keypoints dataset uses 21 keypoints with (x, y, visibility) triplets and one object class named hand.

How to Train YOLOv8 for Hand Pose Estimation

YOLOv8 supports training for pose/keypoint detection, making its pose checkpoints suitable starting points for custom pose tasks.

The general workflow is:

1. prepare the images,
2. annotate each hand,
3. create YOLO Pose labels,
4. configure the dataset YAML,
5. load a YOLOv8 Pose checkpoint,
6. train the model.

Configure the Dataset YAML File

For a 21-keypoint hand dataset, a simplified configuration might look like:

path: /path/to/hand_dataset

train: images/train
val: images/val

kpt_shape: [21, 3]

names:
  0: hand

A complete configuration can also include flip_idx so left-right landmark relationships remain correct during horizontal flipping.

The official Hand Keypoints configuration includes kpt_shape, flip_idx, the hand class, and explicit landmark names.

Select a YOLOv8 Pose Model

YOLOv8 provides several pose checkpoint sizes:

yolov8n-pose.pt
yolov8s-pose.pt
yolov8m-pose.pt
yolov8l-pose.pt
yolov8x-pose.pt

For initial experiments, a smaller model such as:

yolov8n-pose.pt

is useful because it requires fewer computational resources.

Larger models offer greater capacity but normally require more GPU memory and computation.

Start Custom Hand Pose Training

A basic CLI training command is:

yolo pose train model=yolov8n-pose.pt data=hand-keypoints.yaml epochs=100 imgsz=640

The equivalent Python workflow is:

from ultralytics import YOLO

model = YOLO("yolov8n-pose.pt")

model.train(
    data="hand-keypoints.yaml",
    epochs=100,
    imgsz=640
)

YOLOv8 pose models officially support training, validation, inference, and export modes.

Testing a Trained YOLOv8 Hand Pose Model

After training, the best checkpoint can be tested on images, videos, or live camera input.

Visual testing is particularly important for hand pose models because finger landmarks can look incorrect even when general detection metrics appear acceptable.

Run Inference on Images

Using the CLI:

yolo pose predict model=runs/pose/train/weights/best.pt source=hand.jpg

Or with Python:

from ultralytics import YOLO

model = YOLO("runs/pose/train/weights/best.pt")
results = model("hand.jpg")

Keypoints can then be accessed from the returned pose results.

Perform Hand Pose Estimation on Videos

A trained model can also process video:

yolo pose predict model=best.pt source=hand_video.mp4

For every processed frame, the model predicts visible hand instances and their corresponding landmarks.

Video testing is useful for checking whether predictions remain stable when fingers and hands move quickly.

Real-Time Hand Pose Estimation with a Camera

A webcam can be used as an inference source.

For example:

yolo pose predict model=best.pt source=0

In Python:

from ultralytics import YOLO

model = YOLO("best.pt")
results = model(source=0, stream=True)

for result in results:
    keypoints = result.keypoints

Actual real-time performance depends on model size, image resolution, hardware, and the required frame rate.

Improving Hand Pose Estimation Accuracy

Hands are challenging objects for pose estimation because fingers are small, frequently overlap, and can move through a very large number of configurations.

Improving dataset quality is often more valuable than simply increasing the number of training epochs.

Handle Occluded Fingers and Hands

Finger landmarks are often hidden by:

• other fingers,
• held objects,
• sleeves,
• another hand,
• the person’s body.

Include realistic examples of these situations in the training dataset.

Visibility annotations should also be handled consistently so the model receives meaningful supervision for hidden or difficult landmarks.

Use Diverse Hand Positions and Lighting Conditions

Include hands in many orientations.

Useful examples include:

• palm facing camera,
• back of hand,
• side view,
• clenched fist,
• pointing finger,
• spread fingers,
• bent fingers,
• rotated wrist,
• hand holding an object.

Lighting diversity is equally important.

Models trained only on studio-quality images may perform poorly in dark rooms, outdoor scenes, or low-contrast backgrounds.

Improve Keypoint Annotation Quality

Accurate landmark placement is especially important for finger joints because small positional errors can represent large differences in the estimated pose.

Check:

• fingertip locations,
• joint locations,
• keypoint order,
• wrist placement,
• visibility values,
• bounding boxes.

It is useful to visualize annotations before beginning full training.

Applications of YOLOv8 Hand Pose Estimation

Hand pose estimation can support systems that need to understand detailed human hand movement rather than simply detect the presence of a hand.

Ultralytics lists gesture recognition, AR/VR control, robotic manipulation, hand movement analysis, animation, and related uses among the applications of hand-keypoint estimation.

Hand Gesture Recognition

Hand landmarks can be converted into geometric features representing gestures.

For example, an application can analyze:

• finger extension,
• joint angles,
• fingertip distances,
• palm orientation.

These features can then be used to distinguish gestures such as pointing, open palm, fist, or other application-specific commands.

Sign Language Recognition

Hand keypoints can provide useful information for sign-language recognition systems because many signs depend on finger configuration and hand orientation.

However, full sign-language understanding is more complex than static hand pose estimation.

Practical systems may also need:

• temporal motion,
• both hands,
• body pose,
• facial expression,
• contextual sequence information.

YOLOv8 hand landmarks can therefore serve as one visual component in a larger sign-language recognition pipeline.

Human-Computer Interaction

Hand pose estimation can enable touchless control systems.

For example, hand landmarks can be used to detect:

• pointing,
• pinching,
• swiping,
• finger counting,
• virtual button selection.

This can provide a natural interface without requiring a physical mouse or touchscreen.

AR, VR, and Robotics Applications

Hand keypoints are valuable for augmented and virtual reality because the system needs to estimate where the user’s fingers are positioned relative to virtual objects.

Robotics applications can use similar information to analyze human manipulation or guide human-robot interaction.

Ultralytics specifically lists AR/VR controls and robotic manipulation as applications of its Hand Keypoints dataset.

FAQs About YOLOv8 Hand Pose Estimation

Can YOLOv8 perform hand pose estimation?

Yes. YOLOv8 officially supports pose/keypoint detection and its pose variants support training, validation, inference, and export. A YOLOv8 Pose model can therefore be trained or fine-tuned on a hand-keypoint dataset.

How many hand keypoints can YOLOv8 detect?

The number is determined by the dataset and model configuration rather than being permanently fixed.

A common hand layout uses 21 keypoints, and the official Ultralytics Hand Keypoints dataset contains 21 landmarks for every annotated hand.

Custom models can be trained with a different number of landmarks when required.

Does YOLOv8 include a pretrained hand pose model?

Ultralytics provides official YOLOv8 Pose checkpoints such as yolov8n-pose.pt through yolov8x-pose.pt, but these standard YOLOv8 pose checkpoints are general human-pose models rather than dedicated 21-keypoint hand models.

For detailed hand landmarks, train or fine-tune a pose model on a hand-specific dataset. Ultralytics now maintains an official 21-keypoint Hand Keypoints dataset for this purpose, although its current documentation examples use newer YOLO generations.

Can I train YOLOv8 on custom hand keypoints?

Yes. Custom pose training allows you to define your own keypoint layout.

For example, you could train:

kpt_shape: [10, 3]

for a simplified 10-landmark hand representation or:

kpt_shape: [21, 3]

for a detailed 21-landmark structure.

The annotations and YAML configuration must use the same keypoint ordering throughout the dataset.

What dataset format is required for hand pose training?

YOLO Pose training uses labels containing the class, bounding box, and keypoints.

A common format is:

class x_center y_center width height x1 y1 v1 x2 y2 v2 ... xN yN vN

The official Hand Keypoints dataset uses one class (hand) and 21 landmarks represented with x, y, and visibility values.

Can YOLOv8 detect multiple hands at the same time?

Yes. A trained pose model can detect multiple object instances in an image, so a hand-specific model can produce separate bounding boxes and landmark sets for multiple detected hands.

Each hand is treated as an individual detected instance with its own pose output.

Is YOLOv8 suitable for real-time hand pose estimation?

YOLOv8 was designed with an accuracy-speed balance and supports pose inference, making it suitable for real-time or near-real-time applications when the model and hardware are appropriately selected.

Smaller variants such as yolov8n-pose are generally more appropriate when inference speed and limited hardware resources are priorities, while larger models trade additional computation for greater model capacity.

Conclusion

YOLOv8 hand pose estimation combines object localization with detailed keypoint prediction, allowing a model to identify both where a hand appears and how its fingers and joints are positioned.

YOLOv8 officially supports pose/keypoint detection and provides several pose model sizes that can be used as starting points for custom training.

For detailed hand tracking, a 21-keypoint layout provides the wrist plus four landmarks for each of the five fingers. Ultralytics’ official Hand Keypoints dataset follows this structure and contains 26,768 annotated images, one hand class, and 21 (x, y, visibility) landmarks per hand.

By combining high-quality annotations, diverse hand positions, consistent keypoint definitions, and an appropriately sized YOLOv8 Pose model, developers can build hand-pose systems for gesture recognition, sign-language pipelines, human-computer interaction, AR/VR, robotics, and other landmark-based applications.

Jane Torres

I’m Jane Austen, a skilled content writer with the ability to simplify any complex topic. I focus on delivering valuable tips and strategies throughout my articles.