Adding input shift

[BartvanMarrewijk]

Dear Roboflow,

First of all many thanks for making this github open source.
I am using the trackers in a conveyor setting to track objects over time. However, since the conveyor belt speed is not constant both and it is full of objects >200 Bytetrack and SortTracker fail. It incorrectly matches the objects.

However, by adding a simple input U vector, the shift in x or y position both trackers work fine. The shift of the conveyor can be easily determined by for example opencv matched (cv2.matchTemplate)

    def predict(self, u: np.ndarray| None = None) -> None:
        """
        Predict the next state of the bounding box (applies the state transition).
        """
        # Predict state
        if u is not None:
            # Ensure u has the correct shape (n, 1)
            if u.ndim == 1:
                u = u.reshape(-1, 1)
            self.state = (self.F @ self.state).astype(np.float32) + (self.B @ u).astype(np.float32)
        else:
            self.state = (self.F @ self.state).astype(np.float32)
        # Predict error covariance
        self.P = (self.F @ self.P @ self.F.T + self.Q).astype(np.float32)

        # Increase time since update
        self.time_since_update += 1

I think this would be a nice feature, let me know what you think. Maybe a simple wrapper can already help.

[SkalskiP]

Hi @BartvanMarrewijk 👋🏻 Thanks a lot for your interest in Trackers.

Would you be able to share an example video that shows the issue?

The fix you used works, but it feels more like a workaround. We want Trackers to handle cases like this out of the box. If you can share a sample, we can look into a more general and reusable solution.

[BartvanMarrewijk]

Hereby a link to the video with default (Sort) and the custom implementation with input from the template matcher.
https://filesender.surf.nl/?s=download&token=381e1e47-9ab2-4afd-8f5e-737fec7bd37b
(uploading directly to github did not work).

I understand that preferably it handles cases like this out of the box, but with many objects, and a large shift default trackers in my opinion do not work, or I am doing something wrong. The input can be relatively easy determined using cv2. for a more concrete example, please have a look at my fork:
https://github.com/VISNet-framework/trackers/blob/wur_devel/main_example_gar.py

[BartvanMarrewijk]

sorry, another remark. Another example, for 3D data might be interesting as well. In these situations, objects at the back move slowly with respect to objects in the foreground, so tracking them on xyz location + optional box information might better than boxes only.

[SkalskiP]

Hi @BartvanMarrewijk 👋🏻

The reason we lose tracks is that both SORT and ByteTrack use IoU to associate objects between consecutive frames. In your case, the FPS of the video is so low that the tracked objects jump from one position to another between frames. As a result, the overlap is too small and the tracker cannot match them correctly.

Why is the video running at such a low FPS? Is there a way to increase the input video FPS?

[BartvanMarrewijk]

@SkalskiP
Thank you for your time.

I do fully agree that the quality of the video is poor, low image quality and low fps.
However, I don't think you can always or want to improve the FPS. For example, if you are running a setup with multiple camera's, sometimes it is not possible to run with a high fps. A high FPS also means that your deep learning network has to process all those frames, which is not always desirable, GPU load model size etc.

In industrial setup, with a conveyor belt i think tracking by using the input "u" in a kalman filter makes sense. Yes, it is not out of the box, however, it has to possibility to run with a low FPS, enabling a higher throughput and or better deep learning model.

[roel-klein]

I think this would be nice to add.

By default, ByteTrack and SORT use independent Kalman filters for each object, but in many use-cases it is necessary to add global motion compensation.

This global motion can already be known beforehand, or be estimated in a variety of ways.
For example, ORB/SIFT feature matching, optical flow, or cross-correlation based, most already available within opencv.
Examples of this are also included in BOT-SORT (https://github.com/NirAharon/BoT-SORT/blob/main/tracker/gmc.py) and Ultralytics (https://docs.ultralytics.com/reference/trackers/utils/gmc/).

So I think to keep this modular, it would be nice to let the users give an optional global state update in the tracker.update methods.

Some functions or examples could be included for the global motion estimation, but in general this problem does not have a one-size-fits-all solution.

[SkalskiP]

We’re already working on BoT-SORT here: #232.

That’s why I don’t want to merge solutions like this into trackers right away. I’d rather wait until the BoT-SORT work is further along. Then we might be able to connect these ideas and end up with a cleaner approach.