FTC-ML Model Review

[ddiaz]

@Team 12231

Can you look at our account and give pointers?

Sure. I looked at your account, and here are several thoughts:

1. I assume your camera is mounted upside-down on your robot, which is why your training images are all upside down.
   

2. I’m not sure why you have two different “models” that you’re training. Your object is just a cylinder with your team number printed on it, one red and one blue. I would guess that you could just use one label and call it “prop” and label all blue and red objects in videos as “prop”. But if you want different labels and are training different models for each prop that’s fine, but that is expensive (in terms of model training).

3. All of your videos (that I saw, I didn’t open all your videos, you have a ton of them) are just a single static frame - you have 30-60 frames that are all the same image. You never want that, it introduces confirmation bias in the model training because there are so many images that are of the exact same scenario. Each step is only 32 frames, and so you likely go entire training steps with the exact same scenario or even 2 steps with very little variation; the model will begin to converge on that same scenario only to toss out everything it has learned when it hits the next scenario. This will cause the model take much much longer to begin converging on a solution if it ever does. You really should tell the model to ignore 90% of the frames in each video if they’re all the same (you have to mark each frame to ignore).

4. Are you trying to get the model to recognize the numbers only? Understand that TensorFlow does not differentiate colors very well, nor does it differentiate shapes. It is a pattern detector. It may be able to detect dark contrasting numbers on a different contrasting background, but only with a lot of training. Regardless, you need to give the numbers more background in the labeling. Your label is super-tight to the numbers, give them a little breathing room. On the left is how you’re labeling the numbers, on the right is the suggested labeling (I hope this makes sense):
   

5. Your model “Blue12231b” trained after 4,000 steps - and it’s a miracle that it did. If you look at its training metrics, some “miracle” happened after 3,000 steps and it started to converge, and at around 4,000 steps it started to look reasonable. I have no idea how it did that, because the model metrics looked absolutely horrible before that. It’s likely that a combination of “lots of the same scenarios” and “labeling too tightly” caused training issues. Here’s a sample of the training metrics - these metrics show the loss jumping up periodically as the model training becomes confused, the loss should never have been above 2.0 after the first epoch and should then begin converging towards zero (like it started to do after step 3300).
   

6. Your models will show you how well it’s identifying objects in the “model images” - look at them after the model has trained.
   In this example, the model is correctly identifying the object (hooray!)
   

   
   
   In this example, it has no idea what it’s looking at:
   
   
   
   In this example, the training is very off (Identifying random space):
   
   

So how do we improve?

1. If you’re trying to train on the OBJECT (cylinder with numbers) you’re definitely not. If you’re trying to train the model to recognize numbers on a poorly contrasting background, you’re doing that better. If your numbers are dark, choose light colors of the contrasting background.

2. If you’re trying to train on the NUMBERS, you need to give the numbers more surrounding background in your labels. If you’re trying to detect the OBJECT (cylinder) you need to actually surround the object with the label.

3. Consider combining models. Since TensorFlow can’t really differentiate colors or shapes very well, and you’re just looking at the black numbers, there’s really very little difference so long as the model is trained to look at both. Save energy, save time, save frustration.

4. Remove duplicates. Having duplicate images is very very bad. Ignore frames that are flagrant duplicates.

5. LOOK AT YOUR METRICS. If you see graphs that are popping up and down, the model is confused. Things can be “rocky” at the start, but the model should smoothly begin converging toward zero after a short period.

6. LOOK AT YOUR MODEL IMAGES and make sure your model is identifying the object appropriately. If the model isn’t identifying properly, there’s a big problem in your data and you need help.

-Danny