YOLOv8 with DeepSORT combines fast object detection with multi-object tracking. YOLOv8 detects objects in each video frame, while DeepSORT links those detections across frames and assigns persistent tracking IDs. DeepSORT extends SORT by adding deep appearance features, which helps reduce identity switches and improves tracking through short occlusions. Ultralytics itself currently ships trackers such as BoT-SORT and ByteTrack rather than DeepSORT directly, so DeepSORT is typically integrated as a separate tracking library alongside YOLOv8.
Introduction to YOLOv8 with DeepSORT
Object detection tells you what objects are visible in a frame and where they are located. Tracking adds another layer by identifying whether an object seen in the current frame is the same object that appeared in previous frames.
YOLOv8 can handle the detection stage efficiently, while DeepSORT performs the association between detections over time.
This combination is useful for video analytics where simply detecting a person or vehicle is not enough. You may also need to follow that person or vehicle across many frames, count unique objects, analyze movement, or maintain a stable identity while the object moves. DeepSORT was specifically designed to improve SORT by adding appearance-based association.
What Is YOLOv8 with DeepSORT?
YOLOv8 with DeepSORT is a tracking-by-detection pipeline.
YOLOv8 acts as the detector. For every frame, it produces bounding boxes, class predictions, and confidence scores.
DeepSORT receives those detections and decides which current detection corresponds to which existing track.
The final result can contain:
- object class,
- bounding box,
- detection confidence,
- tracking ID,
- movement across frames.
Role of YOLOv8 in Object Detection
YOLOv8 analyzes each frame independently and finds objects belonging to its trained classes.
For example, it might return:
person confidence=0.91
car confidence=0.87
person confidence=0.83
along with a bounding box for each object.
YOLOv8 itself provides the visual detections that DeepSORT needs as input.
Without a detector, DeepSORT does not know which objects are present in the current frame.
Role of DeepSORT in Object Tracking
DeepSORT is responsible for maintaining object identities over time.
For example:
Frame 1:
Person → ID 4
Frame 2:
Person → ID 4
Frame 3:
Person → ID 4
Even though YOLOv8 performs a new detection on every frame, DeepSORT attempts to associate the detections with existing tracks.
It combines motion information with a deep appearance descriptor, which was the major improvement DeepSORT introduced over the original SORT algorithm.
How YOLOv8 and DeepSORT Work Together
The pipeline normally follows three main steps:
Video Frame
↓
YOLOv8 Detection
↓
DeepSORT Association
↓
Tracked Objects + IDs
This process repeats for every frame.
Detecting Objects with YOLOv8
The first step is running YOLOv8 inference.
For example:
from ultralytics import YOLO
model = YOLO("yolov8n.pt")
results = model(frame)
From the results, you can extract:
boxes = results[0].boxes.xyxy
confidences = results[0].boxes.conf
classes = results[0].boxes.cls
These detections then need to be converted into the format expected by the DeepSORT implementation being used.
Passing Detections to DeepSORT
A DeepSORT library typically expects information such as:
bounding box
confidence
class
A common representation is:
[x, y, width, height], confidence, class
The exact API depends on the DeepSORT implementation.
For example, the popular deep-sort-realtime package provides a DeepSORT implementation intended for real-time integration and is based on the original Deep SORT approach.
Assigning Unique Tracking IDs
DeepSORT compares new detections with active tracks.
If a detection is considered to match an existing track:
Existing Track 7
+
New Detection
↓
Track ID 7 continues
If no suitable existing track is found, a new tracking ID may be created.
This allows downstream applications to count unique objects instead of counting the same object again in every frame.
DeepSORT Tracking Process
DeepSORT uses both motion and appearance information.
The main components are:
- Kalman filtering,
- appearance embeddings,
- data association.
Kalman Filter for Motion Prediction
Like SORT, DeepSORT uses a Kalman filter to estimate the likely future state of an object.
Suppose a car has been moving from left to right.
Even before the next detection arrives, the tracker can predict approximately where the car should appear.
This helps reduce incorrect associations between objects.
The original SORT framework uses a Kalman filter and assignment logic for online multi-object tracking.
Appearance Feature Extraction
The major addition in DeepSORT is appearance information.
A deep feature extractor produces an embedding describing how an object looks.
Conceptually:
Detected Person
↓
Appearance Network
↓
Feature Vector
DeepSORT then compares appearance vectors between current detections and previous tracks.
This is particularly useful when objects move close together or temporarily disappear behind another object.
The original DeepSORT paper reports that adding the deep association metric significantly reduced identity switches compared with SORT.
Matching Objects Across Frames
DeepSORT combines motion and appearance information when deciding which detection belongs to which track.
For example:
Track 5:
Expected Position = close
Appearance = similar
Detection A:
Position = close
Appearance = similar
→ likely match
This gives DeepSORT more information than a tracker that relies mainly on box geometry and motion.
How to Set Up YOLOv8 with DeepSORT
Because DeepSORT is not currently one of the built-in tracker choices listed in the official Ultralytics tracking documentation, a common setup is to combine the Ultralytics detector with a separate DeepSORT package. Ultralytics currently documents trackers such as BoT-SORT, ByteTrack, OC-SORT, Deep OC-SORT, FastTracker, and TrackTrack.
Install Required Python Libraries
A common setup is:
pip install ultralytics
pip install deep-sort-realtime
pip install opencv-python
deep-sort-realtime is a maintained real-time-oriented adaptation of Deep SORT available through PyPI.
Then import the main components:
import cv2
from ultralytics import YOLO
from deep_sort_realtime.deepsort_tracker import DeepSort
Load the YOLOv8 Model
Load a pretrained YOLOv8 model:
model = YOLO("yolov8n.pt")
Or load your own trained weights:
model = YOLO("best.pt")
A custom model works the same way as long as its prediction results can be converted into detection inputs for DeepSORT.
Initialize the DeepSORT Tracker
A basic tracker initialization may look like:
tracker = DeepSort(
max_age=30,
n_init=3
)
The available arguments depend on the DeepSORT package version being used.
Parameters commonly control:
- maximum track age,
- number of detections before confirming a track,
- appearance matching,
- embedding configuration.
Run YOLOv8 with DeepSORT on Video
A typical pipeline reads one frame at a time, runs detection, sends detections into DeepSORT, and draws the returned tracks.
Process Video Frames
A simplified example:
import cv2
from ultralytics import YOLO
from deep_sort_realtime.deepsort_tracker import DeepSort
model = YOLO("yolov8n.pt")
tracker = DeepSort(max_age=30)
cap = cv2.VideoCapture("video.mp4")
while True:
success, frame = cap.read()
if not success:
break
results = model(frame, verbose=False)
detections = []
for box in results[0].boxes:
x1, y1, x2, y2 = box.xyxy[0].tolist()
confidence = float(box.conf[0])
class_id = int(box.cls[0])
width = x2 - x1
height = y2 - y1
detections.append(
([x1, y1, width, height], confidence, class_id)
)
tracks = tracker.update_tracks(
detections,
frame=frame
)
cap.release()
The precise accepted input format should always be checked against the DeepSORT library version in your environment.
Track Multiple Objects in Real Time
DeepSORT can maintain multiple tracks simultaneously.
For example:
Person → ID 1
Person → ID 2
Car → ID 3
Car → ID 4
Each object receives an independent track.
This makes the combination suitable for people counting, vehicle analytics, and general multi-object tracking.
Display Bounding Boxes and Tracking IDs
You can draw confirmed tracks with OpenCV:
for track in tracks:
if not track.is_confirmed():
continue
track_id = track.track_id
left, top, right, bottom = track.to_ltrb()
cv2.rectangle(
frame,
(int(left), int(top)),
(int(right), int(bottom)),
(0, 255, 0),
2
)
cv2.putText(
frame,
f"ID: {track_id}",
(int(left), int(top) - 10),
cv2.FONT_HERSHEY_SIMPLEX,
0.6,
(0, 255, 0),
2
)
This produces a video where each tracked object is displayed with a persistent ID.
YOLOv8 with DeepSORT for Custom Object Tracking
YOLOv8 does not need to use standard COCO classes.
A custom YOLOv8 detector can be combined with DeepSORT in the same general way.
Use a Custom-Trained YOLOv8 Model
Load custom weights:
model = YOLO("runs/detect/train/weights/best.pt")
The detector can represent custom objects such as:
helmet
forklift
machine
package
animal
product
DeepSORT then tracks the detections produced by that model.
The tracking system does not require objects to belong to the standard COCO dataset.
Track Selected Object Classes
Sometimes you only need to track certain categories.
For example:
allowed_classes = [0, 2]
for box in results[0].boxes:
class_id = int(box.cls[0])
if class_id not in allowed_classes:
continue
This can reduce unnecessary tracker workload.
For example, a traffic monitoring application might only track:
car
bus
truck
motorcycle
while ignoring other detected classes.
Configure Tracking Parameters
Important tracker parameters can include settings controlling:
- maximum lost-track age,
- track confirmation,
- appearance distance,
- detection confidence,
- embedding model.
These parameters affect how aggressively the tracker preserves identities.
For example, increasing the maximum age may help keep a track alive during temporary occlusion, but it may also increase the chance that an old track is incorrectly matched with a different object.
YOLOv8 DeepSORT Performance and Accuracy
Overall tracking quality depends on both the detector and tracker.
A strong tracker cannot fully compensate for consistently poor detections.
Detection Confidence and Tracking Accuracy
If YOLOv8 misses an object for several frames, DeepSORT has less information available to maintain the track.
Likewise, repeated false detections can create incorrect tracks.
This relationship between detection quality and tracking quality is fundamental to tracking-by-detection systems. The original SORT paper specifically identified detector quality as a major factor affecting tracking performance.
Use a confidence threshold appropriate for your application rather than automatically setting it extremely high.
An overly high threshold may remove useful detections needed to maintain tracks.
Handling Occlusion and Lost Tracks
Appearance features are one reason DeepSORT performs better than plain SORT in many occlusion scenarios.
If an object disappears behind another object and later becomes visible, appearance similarity can help reconnect the detection with the original identity.
DeepSORT was specifically designed to reduce identity switches and improve tracking through longer occlusions by incorporating appearance information.
However, very long occlusions, drastic appearance changes, or crowded scenes can still cause lost tracks and new IDs.
Improving Real-Time Tracking Speed
A YOLOv8 + DeepSORT pipeline performs more work than YOLOv8 detection alone.
Performance can be improved by:
- using
yolov8n.ptor another small detector, - reducing input resolution,
- using GPU inference,
- filtering unnecessary classes,
- processing fewer frames when full frame rate is not required,
- choosing a lightweight appearance embedder.
Appearance feature extraction adds computation, so DeepSORT can be slower than trackers that rely less on Re-ID features.
YOLOv8 with DeepSORT vs Other Trackers
DeepSORT is only one option for multi-object tracking.
Modern YOLO pipelines frequently use SORT-derived methods, ByteTrack, BoT-SORT, and other tracking algorithms.
DeepSORT vs SORT
SORT primarily uses motion and geometry for association.
DeepSORT extends that approach with a learned appearance metric.
Conceptually:
SORT:
Motion + Box Geometry
DeepSORT:
Motion + Box Geometry + Appearance
The extra appearance information can reduce ID switching in difficult situations.
The tradeoff is additional computational cost.
DeepSORT vs ByteTrack
ByteTrack uses a different tracking strategy.
A key idea in ByteTrack is to associate not only high-confidence detections but also lower-confidence detections that may still belong to real tracked objects.
DeepSORT, by contrast, is especially known for incorporating deep appearance information into detection-to-track association.
Ultralytics includes ByteTrack directly as one of its supported trackers, while DeepSORT generally requires separate integration in a standard YOLOv8 setup.
Choosing the Right Tracker for YOLOv8
DeepSORT can be useful when:
- identity consistency is important,
- appearance helps distinguish similar trajectories,
- temporary occlusion is common.
ByteTrack can be a strong choice when:
- you want straightforward Ultralytics integration,
- speed is important,
- reliable detector confidence information is available.
BoT-SORT is another built-in Ultralytics option and can use appearance features as part of its tracking configuration. Ultralytics currently documents BoT-SORT and ByteTrack among its standard tracking options.
Applications of YOLOv8 with DeepSORT
The YOLOv8 and DeepSORT combination is useful wherever individual objects need to be followed across video frames.
People Tracking
A surveillance system can assign unique IDs such as:
Person 1 → ID 14
Person 2 → ID 18
Person 3 → ID 21
These IDs can be used for:
- counting unique people,
- movement analysis,
- entry and exit monitoring,
- trajectory analysis.
Appearance-based association can be especially useful when several people cross paths.
Vehicle Tracking
Vehicles can be tracked across road or parking-area footage.
Possible applications include:
- traffic counting,
- vehicle trajectories,
- speed estimation,
- parking analysis,
- lane monitoring.
A custom YOLOv8 model can also be used when more specific vehicle categories are needed.
Surveillance and Traffic Monitoring
Multi-object tracking allows a system to understand temporal behavior instead of processing every frame independently.
For example:
Vehicle ID 12
Frame 10 → Lane A
Frame 80 → Intersection
Frame 160 → Exit
This enables traffic flow analysis and long-term video analytics.
Sports and Crowd Analysis
Object tracking can be applied to:
- athletes,
- players,
- crowds,
- referees,
- sports equipment.
Tracking IDs make it possible to analyze trajectories and movement patterns over time.
Crowded scenes remain difficult, particularly when many visually similar objects overlap.
Common YOLOv8 DeepSORT Problems
Tracking systems can fail even when detections look visually correct.
The most common problems involve identity consistency.
Duplicate Tracking IDs
A single object may sometimes receive more than one ID over time.
For example:
Frame 20 → Person ID 5
Frame 40 → lost
Frame 50 → Person ID 17
This often happens when the old track expires before the object reappears.
Possible adjustments include:
- increasing track age,
- improving detection consistency,
- improving Re-ID quality,
- lowering overly strict matching settings.
ID Switching Between Objects
An ID switch happens when two objects exchange identities.
For example:
Before crossing:
Person A → ID 3
Person B → ID 7
After crossing:
Person A → ID 7
Person B → ID 3
DeepSORT’s appearance metric was introduced specifically to help reduce this problem compared with SORT.
However, visually similar objects and severe overlap can still cause switches.
Lost Tracks During Occlusion
Objects can disappear behind:
- people,
- vehicles,
- walls,
- poles,
- other scene elements.
The Kalman filter may continue predicting their motion for a limited period.
If the object remains hidden for too long, the track may expire.
Increasing the allowed track age can help in some situations, but it should be balanced against the risk of incorrectly reconnecting unrelated objects.
Slow Tracking Performance
The tracking pipeline includes multiple processing stages:
YOLO inference
+
DeepSORT appearance extraction
+
data association
+
visualization
Any of these can reduce frame rate.
Possible improvements include:
- smaller YOLOv8 model,
- smaller input image,
- GPU acceleration,
- lighter appearance model,
- reduced visualization work,
- tracking only required classes.
FAQs About YOLOv8 with DeepSORT
Can YOLOv8 be used with DeepSORT?
Yes. YOLOv8 can provide object detections to a separate DeepSORT implementation.
However, DeepSORT is not currently listed as one of the built-in tracker configurations in the official Ultralytics Track documentation. Ultralytics currently supports options including BoT-SORT, ByteTrack, OC-SORT, Deep OC-SORT, FastTracker, and TrackTrack.
How does DeepSORT work with YOLOv8?
YOLOv8 detects objects in each video frame.
DeepSORT receives those detections and associates them with existing tracks using motion prediction and appearance features.
The result is a set of objects with persistent tracking IDs.
Does DeepSORT detect objects by itself?
No. DeepSORT is a tracker, not a general-purpose object detector.
It requires detections from another model, such as YOLOv8.
The original DeepSORT implementation is designed to operate on supplied detections and extend SORT with appearance-based association.
Can YOLOv8 with DeepSORT track multiple objects?
Yes.
The pipeline is designed for multi-object tracking.
It can maintain separate tracks for multiple people, vehicles, animals, products, or other objects detected by YOLOv8.
Can I use a custom YOLOv8 model with DeepSORT?
Yes.
A custom YOLOv8 detector can be used as long as its bounding boxes, confidence scores, and class information are converted into the input structure expected by the DeepSORT implementation.
This makes the pipeline suitable for domain-specific tracking.
What is the difference between DeepSORT and ByteTrack?
DeepSORT extends SORT with deep appearance embeddings to improve association and reduce identity switching.
ByteTrack uses a different detection-association strategy that makes use of lower-confidence detections as part of tracking.
ByteTrack is also directly supported in the current Ultralytics tracking interface, whereas DeepSORT commonly requires external integration.
Is YOLOv8 with DeepSORT suitable for real-time tracking?
Yes, it can be suitable for real-time tracking when the detector, appearance model, resolution, and hardware are chosen appropriately.
Smaller YOLOv8 models and GPU acceleration can improve frame rate.
The appearance feature stage adds additional computational cost compared with simpler tracking methods, so actual real-time performance should be benchmarked on the target hardware.
Conclusion
YOLOv8 with DeepSORT creates a tracking-by-detection pipeline where YOLOv8 detects objects and DeepSORT maintains their identities across frames.
DeepSORT builds on SORT’s motion-based tracking by adding a deep appearance metric. This additional visual information can reduce identity switches and improve association when objects move close together or experience short occlusions.
The overall workflow is:
Video
↓
YOLOv8
↓
Bounding Boxes + Classes + Confidence
↓
DeepSORT
↓
Motion + Appearance Matching
↓
Persistent Object IDs
This combination can be used for people tracking, vehicle monitoring, surveillance, traffic analytics, sports analysis, and custom object tracking.
For new Ultralytics projects, it is also worth comparing DeepSORT with the trackers available directly in the current Ultralytics tracking system, particularly ByteTrack and BoT-SORT, since those require less custom integration.
I’m Jane Austen, a skilled content writer with the ability to simplify any complex topic. I focus on delivering valuable tips and strategies throughout my articles.