Oscillation in DetectionBoxes metrics

Thanks for the amazing tool. Could you take a look at team 14140’s “Crash logo test” model and advise. The “DetectionBoxes” graphs exhibit an oscillation between high and low numbers and then zeros out as the training wrapped up. Based on the numbers and eval images, it looks like the model was actually working pretty well around step 400, then things start to go off the rails. We not quite sure how to interpret this or which direction to take this. Thanks!!

image

Since you gave me permission to view your account details, I looked at your account content. We don’t (yet) track what videos are used to compile into datasets, but since there are only a few videos in your account and you labeled them nicely it wasn’t a difficult task to determine which ones I should be looking at (so thank you for that!).

In your videos, you have a multitude of mis-labeled frames that would make it extremely difficult for any model to approximate accurate object recognition, which compounds into near-impossible difficulty in predicting correct detection boxes. The training process requires “consistent” labeling of objects, which includes bounding box drawing. When the object isn’t able to be completely represented - for instance the object is half out of the frame or when the object is occluded by another object - then you should faithfully indicate what is part of the object and what isn’t. Unfortunately your labeled frames are at times wildly inconsistent which leads to issues in training (depending on what order the training images are randomized into during a given epoch); if inconsistently-labeled frames all appear clustered at the beginning or spread out within the epoch then the model can usually recover, but if they’re clustered at the end of the epoch the model will often enough fall flat on its face during a given epoch/checkpoint because it becomes confused by what it’s seeing.

What I think you did RIGHT:
(1) I love your zoom in/out video - that’s FANTASTIC. However, you need to control the bounding boxes so they’re consistently around the object (and to the borders when the object goes out of frame).
(2) You have a lot of different twisting/off-angle scenarios, that’s great. Your object won’t always be seen as a perfect circle, and so the model is being trained with perspective.

What I think you need to fix:
(1) You have a LOT of inconsistencies in how your bounding boxes are labeling the object. Both in the size of the bounding box (the pixel distance between the edges of the object to the edge of the box) and in how the bonding box is labeling the object (your bounding box should never be “inside” your object when the object isn’t being occluded or out-of-frame).

Here are a few examples of what I’m seeing:

Video: Crash Test Logo up down
Frame: 168, 242
Description: Your bounding box on the left and right are bounding the inner yellow/black, but on the top/bottom it’s bounding the outside white border too.

Video: Crash Test Logo up down
Frame: 53, 379
Description: This frame is completely unlabeled, which tells the system that the object doesn’t exist in the frame and so everything in the frame should be considered “background”. If you wish to ignore these frames, make sure the “Ignore this frame” is checked like you did for Frame 380.

Video: Crash Test Logo side to side
Frame: 1
Description: Notice the difference in the distance from the object to the edges of the bounding boxes between Frame 21 of “Crash Test Logo up down” and this frame. The difference is like the grand canyon - but it’s still acceptable (just your metrics for bounding boxes might be lower than it could be). HOWEVER, once you get to frame 135 things start going unacceptably off the rails and your bounding box is horrible again until frame 219. Spend some time cleaning up those bounding boxes!

Video: Crash test Logo Zoom In/Out
Frame: 166
Description: What’s the object? The bounding box isn’t around the white border of the object. It isn’t around the yellow/black circle of the object. It’s INSIDE the middle of the object. That is not okay. Manually resize your bounding boxes to go around the full object (white border, leaving some “background” border around the object).

I hope this helps.
-Danny

Awesome. This is just what the kids needed. Much gratitude for the patience with the newbies!

One last thing - @lizlooney reminded me that once you update your bounding boxes you need to regenerate your dataset (I highly recommend deleting your current data set ASAP since we know it has issues). The dataset contains all of the frame and bounding box data to train the model, but doesn’t automatically update when you update the video labels. Think of the dataset as a “complete snapshot” for training a model.

Let me know how it turns out!

Thanks! That would have been an easy trap to fall into. We will make sure we delete the flawed model and dataset as soon as we are done learning from them and BEFORE we try again.