Change in motion of object and 1 missed det led to change in tracking ID. about boxmot

[edblu1]

Is there any way I can test if it is Kalman Filter issue or not?

What helped me debug the behavior of the KF is to also return the KF bbox predictions from the update method and then also plot them on the video output, this way you would see what the Kalman Filter is doing during those phases of missed detections and if this is the issue :)

[mikel-brostrom]

You have an evolve.py script in this repo 🚀

[alaap001]

+1
I forgot to mention this is happening for DeepOCSort and HybridSort, but BotSort is working fine. Could it be the way ReID is being used in both?

[mikel-brostrom]

I would say it is probably more of a Kalman Filter configuration issue. What is your FPS on the camera?

[alaap001]

it is 15 FPS,

Is there any way I can test if it is Kalman Filter issue or not?

[mikel-brostrom]

You could try to scale the process noise covariance. In DeepCOSORT these lines would look like this:

        self.kf.Q[4:6, 4:6] *= self.Q_xy_scaling
        self.kf.Q[-1, -1] *= self.Q_s_scaling

This is because the predictions can be too strict and stiff for your specific use-case, especially if it is a dynamic environment where the bounding box sizes and positions change a lot.

[alaap001]

Ok so if I understood you correctly, Kalman filter has 2 noise factors, Q and R, where Q represents uncertainty in model prediction.
And we are trying to scale Q because my model bbox can change cz of dynamic env where changes can be abrupt and sudden.

I tried to make changes but Q_xy_scaling and Q_s_scaling are not defined
AttributeError: 'KalmanBoxTracker' object has no attribute 'Q_xy_scaling'

Can you help me understand How should these be defined?

for testing, I tried different values fro scaling from 0.01 to 0.05, 0.5 to even 1.0. All resulted in the same output

Thanks a lot for the help.

[mikel-brostrom]

I tried to make changes but Q_xy_scaling and Q_s_scaling are not defined
AttributeError: 'KalmanBoxTracker' object has no attribute 'Q_xy_scaling'

Yup, this is not implemented so you would need to find your way. To start with, you could just define them as class variables and play around with them

[mikel-brostrom]

If you want to know more check out this:

#1484

[alaap001]

Thanks for the reference,
Ya I've been playing around with a small range and let's see what comes up.

1 Question, isn't Kalman filter covarience working the same for both DeepOCSort and BOTSort?
If it was Kalman, Why would it work with BotSort? (Altho in BotSort tracker loses track as soon as human goes out of frame even after increasing track_buffer to 600) But that's a sep issue.

[dalerxli]

您好！您的邮件已收到，谢谢。

[alaap001]

Is there any way I can test if it is Kalman Filter issue or not?

What helped me debugging the behaviour of the KF is to also return the KF bbox predictions from the update method and then also plot them on the video output, this way you would see what the Kalman Filter is doing during those phases of missed detections and if this is the issue :)

This sounds like a good idea, I tried different covariance values, but unfortunately, that didn't help fix the problem. Let me try plotting bbox and see if that helps.

Will get back here in sometime, if anyone has already done something similar to debug these tracking issues, please tag in this issue, it'll be great to have useful resources.

Thanks for the help. :)

[mikel-brostrom]

What helped me debugging the behaviour of the KF is to also return the KF bbox predictions from the update method and then also plot them on the video output

Yup, this is a good recommendation

[alaap001]

Thanks a lot for the suggestion, I tried what you asked and plotted Kalman filter and changed values of Q matrix to see impact, even tried Q to be an identity matrix.

Here are the problems I noticed with KF:

Issue

In this blue box is of KF and box with id: 3 is old tracked id and id: 5 is FP switch of id.

Here as we can see, id: 3 was the original id being tracked, but as soon as a human walked off the van and we missed 1 det in between, the new det came in and got assigned a diff id: 5 instead of same. The KF of the original id: 3 is expecting him to move ahead but he moved back and a new id got assigned even though overlap is significant here.

Simlarly

In this blue box is of KF and box with id: 2 is old tracked id and id: 3 is FP switch of id.

Note in the above image id: 2 was the original track id. Later after 2,3 missed dets we got a new detection which is of same person but got assigned a different id. At this point KF of old det is way off the original path but still has some degree of overlap.

Is there anything we can do in tracking algo to fix issues like these in an environment where movement is mostly unpredictable??

Using identity matrix and changing scale factor for Q has helped in couple of cases but not in the above 2 where a new det (with missed dets in btw) get's assigned a new id for the same object.

PS: A good article to understand KF better: The Kalman Filter: An algorithm for making sense of fused sensor insight
Thanks. :)

[mikel-brostrom]

Multiple predictions ahead could help here

[alaap001]

Multiple predictions ahead could help here

Hey, I'm sorry, I didn't get you. Do you mean a better detector?

[mikel-brostrom]

Not sure what value you set max_age to, but the number of predictions ahead made when the track is lost is exactly max_age. During this period, the tracker relies solely on its prediction model without any new observations to update its state.

boxmot/boxmot/trackers/deepocsort/deep_ocsort.py

Lines 163 to 165 in dccde97

	else:
	self.kf.update(det)
	self.frozen = True

[alaap001]

Oh ok,
here are my params:

tracker = DeepOCSORT(
    model_weights = Path('clip_market1501.pt'),  # which ReID model to use
    device = 'cuda:0',
    det_thresh = 0.3,
    min_hits = 3,
    w_association_emb=0.7,
    iou_threshold = 0.2,
    # embedding_off = False,
    fp16 = False,
    # max_obs = 90,
    inertia=0.4,
    max_age=60,
    # cmc_off = False,
    # aw_off = False,
    # alpha_fixed_emb = 0.95, 
    # per_class=False,
    # new_kf_off = False 
)

I had set it to 60 hoping that we would maintain the track even if the human is hidden for 3 to 4 seconds (15FPS video)

[mikel-brostrom]

I had set it to 60 hoping that we would maintain the track even if the human is hidden for 3 to 4 seconds (15FPS video)

That is right. It keeps it, but it is lost, although still making predictions for it... And lives as lost for 60 frames before removed

[mikel-brostrom]

The easiest solution for you IMO, would be to do a hyperparameter search. There are MANY parameters in these trackers and it is usually not trivial to figure out what is going wrong.

[alaap001]

Ya, I was thinking of using RayTune. Do you have any better suggestions based on your experience for this?

Also, do you recommend training of ReID model? I was honestly hoping ReID to recognize that this is the same object as shown in image 1 but somehow it didn't.

[alaap001]

@mikel-brostrom

do you think we can use something like BPReid to make the tracking more robust?
How hard it'll be to integrate this, and is it worth it to improve DeepOCSort performance?

[github-actions]

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.
Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!